Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add metadata_log_entries table to Iceberg #20410

Merged
merged 2 commits into from
Mar 25, 2024
Merged

Add metadata_log_entries table to Iceberg #20410

merged 2 commits into from
Mar 25, 2024

Conversation

oneonestar
Copy link
Member

@oneonestar oneonestar commented Jan 18, 2024 •
edited by ebyhr
Loading

Description

Add metadata_log_entries table to Iceberg

SELECT * FROM "test_table$metadata_log_entries"

             timestamp                 |                                                              file                                                          | latest_snapshot_id  | latest_schema_id | latest_sequence_number
---------------------------------------+----------------------------------------------------------------------------------------------------------------------------+---------------------+------------------+------------------------
 2024-01-16 15:55:31.172 Europe/Vienna | hdfs://hadoop-master:9000/user/hive/warehouse/test_table/metadata/00000-39174715-be2a-48fa-9949-35413b8b736e.metadata.json | 1221802298419195590 |                0 |                      1
 2024-01-16 17:19:56.118 Europe/Vienna | hdfs://hadoop-master:9000/user/hive/warehouse/test_table/metadata/00001-e40178c9-271f-4a96-ad29-eed5e7aef9b0.metadata.json | 7124386610209126943 |                0 |                      2

Additional context and related issues

Reference:
https://iceberg.apache.org/docs/latest/spark-queries/#metadata-log-entries
https://www.dremio.com/blog/getting-started-with-flink-sql-and-apache-iceberg/#h-metadata-log-entries

Release notes

(x) Release notes are required, with the following suggested text:

# Iceberg
* Add support for `metadata_log_entries` system table. ({issue}`20410`)

@cla-bot cla-bot bot added the cla-signed label Jan 18, 2024
@github-actions github-actions bot added docs iceberg Iceberg connector labels Jan 18, 2024
@oneonestar oneonestar force-pushed the ICEBERG_METADATA_LOG_ENTRIES branch 3 times, most recently from b0f67c5 to 4a04f0e Compare January 19, 2024 04:52
@findinpath findinpath self-requested a review January 22, 2024 05:28
@oneonestar oneonestar requested a review from findinpath January 29, 2024 01:31
@oneonestar oneonestar mentioned this pull request Feb 5, 2024
Closed
@jkylling jkylling self-requested a review February 5, 2024 20:23
jkylling
jkylling reviewed Feb 6, 2024
View reviewed changes
Copy link
Contributor

@jkylling jkylling left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for adding this!

...rino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergMetastoreAccessOperations.java Outdated Show resolved Hide resolved
plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorTest.java Outdated Show resolved Hide resolved
plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/MetadataLogEntriesTable.java Show resolved Hide resolved
plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/MetadataLogEntriesTable.java Show resolved Hide resolved
...c/test/java/io/trino/plugin/iceberg/catalog/glue/TestIcebergGlueCatalogAccessOperations.java Show resolved Hide resolved
@oneonestar oneonestar force-pushed the ICEBERG_METADATA_LOG_ENTRIES branch 2 times, most recently from 02d37e7 to 63c7633 Compare February 7, 2024 13:27
@oneonestar oneonestar requested a review from jkylling February 8, 2024 00:11
@oneonestar oneonestar force-pushed the ICEBERG_METADATA_LOG_ENTRIES branch 2 times, most recently from 575ba33 to 3284154 Compare February 17, 2024 23:58
@findinpath findinpath mentioned this pull request Feb 19, 2024
Closed
findinpath
findinpath approved these changes Feb 20, 2024
View reviewed changes
Copy link
Contributor

@findinpath findinpath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM % comments

@findinpath
Copy link
Contributor

@ebyhr could you please run this PR with secrets ?
I see changes in TestIcebergGlueCatalogAccessOperations.

@ebyhr
Copy link
Member

ebyhr commented Feb 20, 2024

/test-with-secrets sha=328415474d00cd4b90a29150616b84bde9f445f2

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 20, 2024 •
edited
Loading

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/7972684667

@ebyhr ebyhr force-pushed the ICEBERG_METADATA_LOG_ENTRIES branch from 3284154 to 8aa32a6 Compare February 20, 2024 13:30
@ebyhr
Copy link
Member

ebyhr commented Feb 20, 2024

/test-with-secrets sha=8aa32a6a8bdb63d586c9fe3179a6ab0980c928e6

@ebyhr
Copy link
Member

ebyhr commented Feb 20, 2024

Rebased on master to fix CI failure with secrets.

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 20, 2024 •
edited
Loading

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/7975058140

@ebyhr ebyhr self-requested a review February 20, 2024 22:43
@oneonestar oneonestar force-pushed the ICEBERG_METADATA_LOG_ENTRIES branch from 8aa32a6 to 464b7db Compare February 22, 2024 02:01
@oneonestar oneonestar force-pushed the ICEBERG_METADATA_LOG_ENTRIES branch 2 times, most recently from 8d3c144 to 44bb855 Compare March 11, 2024 07:33
@oneonestar
Copy link
Member Author

Rebased to master and resolved conflicts.

ebyhr
ebyhr reviewed Mar 11, 2024
View reviewed changes
plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorTest.java Outdated Show resolved Hide resolved
plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergSystemTables.java Outdated Show resolved Hide resolved
plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergSystemTables.java Outdated Show resolved Hide resolved
plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergSystemTables.java Outdated Show resolved Hide resolved
plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergSystemTables.java Outdated Show resolved Hide resolved
plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/MetadataLogEntriesTable.java Outdated Show resolved Hide resolved
plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorTest.java Outdated Show resolved Hide resolved
plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorTest.java Outdated Show resolved Hide resolved
plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergSystemTables.java Outdated Show resolved Hide resolved
...rino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergMetastoreAccessOperations.java Outdated Show resolved Hide resolved
@oneonestar oneonestar force-pushed the ICEBERG_METADATA_LOG_ENTRIES branch from 7dc18a1 to d7547f1 Compare March 12, 2024 01:16
@oneonestar oneonestar requested a review from ebyhr March 12, 2024 01:24
@ebyhr ebyhr force-pushed the ICEBERG_METADATA_LOG_ENTRIES branch from d7547f1 to c9aecce Compare March 12, 2024 01:28
@oneonestar oneonestar force-pushed the ICEBERG_METADATA_LOG_ENTRIES branch 5 times, most recently from f41d0e8 to 1f8851a Compare March 12, 2024 06:34
@oneonestar
Copy link
Member Author

OPTIMIZE creates two metadata log entries since 947f972

Updated the test accordingly at https://github.com/trinodb/trino/pull/20410/files#diff-2c77e347c613ee81e28953f6b552f710e44bb21e3f75b140770226ac7a1647ceR239-R251

@oneonestar oneonestar requested a review from ebyhr March 12, 2024 07:30
@ebyhr
Copy link
Member

ebyhr commented Mar 18, 2024

@oneonestar Could you rebase on master as #21034 (Iceberg 1.5.0 bump) landed. Sorry, I missed merging this PR.

@oneonestar oneonestar force-pushed the ICEBERG_METADATA_LOG_ENTRIES branch from 1f8851a to f4337bb Compare March 18, 2024 01:43
@oneonestar
Copy link
Member Author

Rebased and updated tests. latest_snapshot_id is no longer null after RTAS.

@oneonestar
Copy link
Member Author

The error seems unrelated.

@ebyhr ebyhr force-pushed the ICEBERG_METADATA_LOG_ENTRIES branch from f4337bb to d673332 Compare March 25, 2024 02:51
@ebyhr ebyhr merged commit 8cec8cd into trinodb:master Mar 25, 2024
25 of 46 checks passed
@github-actions github-actions bot added this to the 444 milestone Mar 25, 2024
@mosabua mosabua mentioned this pull request Mar 25, 2024
Merged
@colebow colebow mentioned this pull request Mar 27, 2024
Closed
@findinpath findinpath mentioned this pull request Mar 29, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@findinpath findinpath findinpath approved these changes

@jkylling jkylling jkylling approved these changes

@ebyhr ebyhr ebyhr approved these changes

Assignees
No one assigned
Labels
cla-signed docs iceberg Iceberg connector
Milestone
444
Development

Successfully merging this pull request may close these issues.
4 participants
@oneonestar @findinpath @ebyhr @jkylling