Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for v2Checkpoint in Delta Lake #19507

Merged
merged 2 commits into from
Mar 29, 2024
Merged

Add support for v2Checkpoint in Delta Lake #19507

merged 2 commits into from
Mar 29, 2024

Conversation

ebyhr
Copy link
Member

@ebyhr ebyhr commented Oct 24, 2023

Description

Relates to #19345

Release notes

(x) Release notes are required, with the following suggested text:

# Delta Lake
* Add support for reading tables with [v2 checkpoint](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#v2-spec). ({issue}`19345`)

@cla-bot cla-bot bot added the cla-signed label Oct 24, 2023
@ebyhr ebyhr self-assigned this Oct 24, 2023
@github-actions github-actions bot added the delta-lake Delta Lake connector label Oct 24, 2023
@ebyhr
Copy link
Member Author

ebyhr commented Nov 6, 2023

Suspending this work to prioritize #19588

@ebyhr ebyhr force-pushed the ebi/delta-v2-checkpoint branch from 385853b to ad85a42 Compare November 17, 2023 01:01
@ebyhr ebyhr mentioned this pull request Jan 22, 2024
Merged
@ebyhr ebyhr force-pushed the ebi/delta-v2-checkpoint branch 3 times, most recently from 863d65a to f182fdf Compare February 19, 2024 05:17
@ebyhr ebyhr marked this pull request as ready for review February 19, 2024 05:18
@findinpath findinpath self-requested a review February 21, 2024 08:40
@ebyhr ebyhr force-pushed the ebi/delta-v2-checkpoint branch 2 times, most recently from f099646 to 5b87fe4 Compare February 22, 2024 03:56
@ebyhr
Copy link
Member Author

ebyhr commented Feb 22, 2024

Rebased on master to resolve logical conflicts.

@ebyhr ebyhr force-pushed the ebi/delta-v2-checkpoint branch from 5b87fe4 to 06143f1 Compare February 27, 2024 01:37
@ebyhr ebyhr requested a review from alexjo2144 February 27, 2024 01:42
findinpath
findinpath reviewed Feb 27, 2024
View reviewed changes
...lta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/CheckpointMetadataEntry.java Show resolved Hide resolved
...elta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/DeltaLakeSchemaSupport.java Outdated Show resolved Hide resolved
...in/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/SidecarEntry.java Show resolved Hide resolved
...n/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TableSnapshot.java Show resolved Hide resolved
...c/main/java/io/trino/plugin/deltalake/transactionlog/checkpoint/CheckpointSchemaManager.java Show resolved Hide resolved
...ke/src/main/java/io/trino/plugin/deltalake/transactionlog/checkpoint/TransactionLogTail.java Show resolved Hide resolved
...n/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TableSnapshot.java Outdated Show resolved Hide resolved
...n/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TableSnapshot.java Outdated Show resolved Hide resolved
plugin/trino-delta-lake/src/test/resources/deltalake/v2_checkpoint_parquet/README.md Show resolved Hide resolved
@ebyhr ebyhr force-pushed the ebi/delta-v2-checkpoint branch from 06143f1 to e266c65 Compare February 29, 2024 07:22
@ebyhr ebyhr requested a review from findepi February 29, 2024 07:22
findinpath
findinpath reviewed Feb 29, 2024
View reviewed changes
Copy link
Contributor

@findinpath findinpath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM % small comments

...o-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/DomainMetadataEntry.java Outdated Show resolved Hide resolved
...o-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/DomainMetadataEntry.java Outdated Show resolved Hide resolved
...o-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/DomainMetadataEntry.java Outdated Show resolved Hide resolved
...in/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/V2Checkpoint.java Show resolved Hide resolved
...n/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TableSnapshot.java Outdated Show resolved Hide resolved
...n/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TableSnapshot.java Outdated Show resolved Hide resolved
plugin/trino-delta-lake/src/test/resources/databricks133/v2_checkpoint_json/README.md Show resolved Hide resolved
...n/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TableSnapshot.java Outdated Show resolved Hide resolved
...in/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/V2Checkpoint.java Show resolved Hide resolved
...nt_json/_delta_log/00000000000000000001.checkpoint.ae4ea00c-afe0-4e8b-ad87-28125262bd44.json Show resolved Hide resolved
@ebyhr ebyhr force-pushed the ebi/delta-v2-checkpoint branch from e266c65 to 2c4ab27 Compare March 1, 2024 10:26
@ebyhr ebyhr force-pushed the ebi/delta-v2-checkpoint branch from 2c4ab27 to b7296fb Compare March 1, 2024 10:28
@github-actions github-actions bot added the docs label Mar 1, 2024
@findinpath
Copy link
Contributor

In case if #20851 lands first, please add as well a test in TestDeltaLakeAlluxioCacheFileOperations

@ebyhr
Copy link
Member Author

ebyhr commented Mar 23, 2024

Rebased on master to resolve conflicts.

@ebyhr ebyhr requested a review from Praveen2112 March 25, 2024 23:22
@ebyhr ebyhr force-pushed the ebi/delta-v2-checkpoint branch from 9f1a3e4 to 72a44ac Compare March 28, 2024 02:00
@ebyhr
Copy link
Member Author

ebyhr commented Mar 28, 2024

Addressed comments.

@findinpath findinpath requested review from pajaks and wendigo March 28, 2024 13:34
@wendigo
Copy link
Contributor

wendigo commented Mar 28, 2024

@ebyhr merge at will :)

@ebyhr ebyhr force-pushed the ebi/delta-v2-checkpoint branch from 72a44ac to 6713bd1 Compare March 29, 2024 00:22
@ebyhr ebyhr merged commit f1773d4 into trinodb:master Mar 29, 2024
5 of 15 checks passed
@ebyhr ebyhr deleted the ebi/delta-v2-checkpoint branch March 29, 2024 00:22
@ebyhr ebyhr mentioned this pull request Mar 29, 2024
Open
2 tasks
@github-actions github-actions bot added this to the 444 milestone Mar 29, 2024
@mosabua mosabua mentioned this pull request Mar 29, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pajaks pajaks pajaks approved these changes

@findinpath findinpath findinpath approved these changes

@Praveen2112 Praveen2112 Praveen2112 approved these changes

@wendigo wendigo wendigo approved these changes

@alexjo2144 alexjo2144 Awaiting requested review from alexjo2144

@findepi findepi Awaiting requested review from findepi

Assignees

@ebyhr ebyhr

Labels
cla-signed delta-lake Delta Lake connector docs
Milestone
444
Development

Successfully merging this pull request may close these issues.
5 participants
@ebyhr @findinpath @wendigo @Praveen2112 @pajaks