Support Iceberg refs system table #15649

jackye1995 · 2023-01-10T07:18:03Z

Description

Support refs system table in Iceberg

Additional context and related issues

Users can query the system table with:

SELECT * FROM "table$refs"

The schema is the same as the one in Iceberg repo: https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/RefsTable.java

Release notes

( ) This is not user-visible or docs only and no release notes are required.
(x) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text:

alexjo2144

How do you feel about calling the table REFERENCES spelled out?

This also needs to be added to iceberg.rst for the docs here: https://trino.io/docs/current/connector/iceberg.html#metadata-tables

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/RefsTable.java

jackye1995 · 2023-01-13T17:49:38Z

How do you feel about calling the table REFERENCES spelled out?

I am using REFS because that is what Iceberg core library uses, which means Spark and Flink system tables are also names as that. But if you think REFERENCES makes more sense in Trino, happy to name it that way.

findinpath · 2023-01-14T06:34:39Z

This also needs to be added to iceberg.rst for the docs here: https://trino.io/docs/current/connector/iceberg.html#metadata-tables

See for reference: https://github.com/apache/iceberg/blob/master/format/spec.md#snapshot-reference

jackye1995 · 2023-01-17T19:38:39Z

@alexjo2144 updated everything except the name REFS, let me know if I need to name it to REFERENCES

findinpath · 2023-01-18T08:11:12Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/RefsTable.java

+            }
+
+            if (ref.minSnapshotsToKeep() != null) {
+                pagesBuilder.appendInteger(ref.minSnapshotsToKeep());


You could add in PageListBuilder new methods:

appendInteger(Integer)

appendBigint(Long)

to encapsulate the null check and make the code in this class more concise.

Added new methods as suggested

alexjo2144 · 2023-01-18T17:52:36Z

How do you feel about calling the table REFERENCES spelled out?

I am using REFS because that is what Iceberg core library uses, which means Spark and Flink system tables are also names as that. But if you think REFERENCES makes more sense in Trino, happy to name it that way.

@ebyhr or @findepi either of you have thoughts on this? Matching the other engines vs spelling the whole word out.

alexjo2144

Looks good to me, pending input from maintainers on the table name.

alexjo2144 · 2023-01-18T17:54:57Z

docs/src/main/sphinx/connector/iceberg.rst

+``$refs`` table
+^^^^^^^^^^^^^^^^^^^^^


I think these are supposed to match width. @mosabua might want to take a look at the docs addition, but thanks for adding it

Updated to match the width

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/util/PageListBuilder.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/RefsTable.java

github-actions · 2023-02-20T20:54:05Z

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

bitsondatadev · 2023-02-21T20:33:06Z

@jackye1995, would you mind replying to the last comments here please?

@ebyhr %comments, is this ready to be merged?

jackye1995 · 2023-02-23T19:10:06Z

@ebyhr sorry did not notice you have one more comment, updated. Let me know if there is anything else needed.

bitsondatadev · 2023-02-23T19:15:40Z

@ebyhr sorry did not notice you have one more comment, updated. Let me know if there is anything else needed.

Thanks @jackye1995! ~~I think there are a few other small naming fixes in #15646 if you can address those as well please!~~ nvm you fixed it already!

After that @ebyhr should take a look shortly! You rock!

electrum · 2023-02-24T02:24:44Z

I think it's fine to match the other implementations, especially since the intent here is to have the same schema.

jackye1995 · 2023-02-24T19:59:30Z

@electrum @ebyhr @findepi is there any plan to merge this and the related PRs for Iceberg reference support? We are about to cut Iceberg 1.2 branch, and have achieved all the same supports in Spark and Flink, would be great to have feature parity in Trino.

ebyhr · 2023-03-08T23:47:51Z

Could you rebase on master to resolve conflicts?

jackye1995 · 2023-03-09T00:14:37Z

Can I get a confirmation if this PR is ever gonna be merged once rebased? If not I will just go ahead to apply the patch internally and you can close this PR instead of me continuing to rebase.

bitsondatadev · 2023-03-09T00:41:23Z

@jackye1995 I'm meeting with @electrum tomorrow I'll ask him to respond. I really want this one to land!

bitsondatadev · 2023-03-09T01:40:03Z

@jackye1995 would you also be able to do this change suggested by David to go back to REFS instead of REFERENCES?

https://trinodb.slack.com/archives/CP1MUNEUX/p1678319193961979?thread_ts=1677280433.219989&channel=CP1MUNEUX&message_ts=1678319193.961979

I will also update you on what David says about how close this is to merge.

ebyhr · 2023-03-09T01:46:46Z

would you also be able to do this change suggested by David to go back to REFS instead of REFERENCES?

The current implementation is already refs.

bitsondatadev · 2023-03-09T01:51:16Z

would you also be able to do this change suggested by David to go back to REFS instead of REFERENCES?

The current implementation is already refs.

Ooops 🫣🫠

jackye1995 · 2023-03-09T17:18:49Z

Close PR, this will be available in AWS Athena by 3/31 in all AWS regions.

electrum · 2023-03-09T17:40:25Z

Sorry for the confusion on this. I'll merge after the test passes. Thanks for your work on this!

bitsondatadev · 2023-03-12T12:34:39Z

@electrum looks like the tests passed! Are we ready to merge this?

electrum · 2023-03-14T05:07:24Z

There was another conflict. Rebased again.

findepi · 2023-03-16T11:13:41Z

Per @ebyhr 's #16576 it seems the PR broke master builds.
@jackye1995 can you please help us and remind to /test-with-secrets?

cla-bot bot added the cla-signed label Jan 10, 2023

jackye1995 requested review from findepi, findinpath and alexjo2144 January 10, 2023 07:18

alexjo2144 reviewed Jan 13, 2023

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/RefsTable.java Outdated Show resolved Hide resolved

jackye1995 force-pushed the ref-table branch from 09dfb87 to cb111a7 Compare January 17, 2023 19:37

jackye1995 requested a review from alexjo2144 January 17, 2023 19:38

github-actions bot added the docs label Jan 17, 2023

jackye1995 force-pushed the ref-table branch from cb111a7 to 3cf0e25 Compare January 17, 2023 19:45

findinpath reviewed Jan 18, 2023

View reviewed changes

alexjo2144 approved these changes Jan 18, 2023

View reviewed changes

jackye1995 force-pushed the ref-table branch from 3cf0e25 to e68b82b Compare January 18, 2023 18:18

ebyhr approved these changes Jan 27, 2023

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/util/PageListBuilder.java Show resolved Hide resolved

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/RefsTable.java Show resolved Hide resolved

github-actions bot added the stale label Feb 20, 2023

github-actions bot removed the stale label Feb 21, 2023

ebyhr mentioned this pull request Feb 21, 2023

Support Iceberg version travel by reference name #15646

Closed

jackye1995 force-pushed the ref-table branch from e68b82b to 864d1e2 Compare February 23, 2023 19:09

jackye1995 force-pushed the ref-table branch 2 times, most recently from 3b429eb to 72e6606 Compare February 23, 2023 19:20

jackye1995 closed this Mar 9, 2023

electrum reopened this Mar 9, 2023

electrum approved these changes Mar 9, 2023

View reviewed changes

github-actions bot added the iceberg Iceberg connector label Mar 9, 2023

Support Iceberg refs system table

3d9316f

electrum force-pushed the ref-table branch from 6db391b to 3d9316f Compare March 14, 2023 05:06

kokosing force-pushed the master branch from 3f05134 to 58d6356 Compare March 14, 2023 11:34

electrum merged commit 1ad04ba into trinodb:master Mar 15, 2023

github-actions bot added this to the 411 milestone Mar 15, 2023

colebow mentioned this pull request Mar 15, 2023

Add Trino 411 release notes #16552

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Iceberg refs system table #15649

Support Iceberg refs system table #15649

jackye1995 commented Jan 10, 2023

alexjo2144 left a comment

jackye1995 commented Jan 13, 2023

findinpath commented Jan 14, 2023

jackye1995 commented Jan 17, 2023

findinpath Jan 18, 2023

jackye1995 Jan 18, 2023

alexjo2144 commented Jan 18, 2023

alexjo2144 left a comment

alexjo2144 Jan 18, 2023

jackye1995 Jan 18, 2023

github-actions bot commented Feb 20, 2023

bitsondatadev commented Feb 21, 2023

jackye1995 commented Feb 23, 2023

bitsondatadev commented Feb 23, 2023 •

edited

Loading

electrum commented Feb 24, 2023

jackye1995 commented Feb 24, 2023

ebyhr commented Mar 8, 2023

jackye1995 commented Mar 9, 2023

bitsondatadev commented Mar 9, 2023 •

edited

Loading

bitsondatadev commented Mar 9, 2023

ebyhr commented Mar 9, 2023

bitsondatadev commented Mar 9, 2023

jackye1995 commented Mar 9, 2023

electrum commented Mar 9, 2023

bitsondatadev commented Mar 12, 2023

electrum commented Mar 14, 2023

findepi commented Mar 16, 2023

		``$refs`` table
		^^^^^^^^^^^^^^^^^^^^^

Support Iceberg refs system table #15649

Support Iceberg refs system table #15649

Conversation

jackye1995 commented Jan 10, 2023

Description

Additional context and related issues

Release notes

alexjo2144 left a comment

Choose a reason for hiding this comment

jackye1995 commented Jan 13, 2023

findinpath commented Jan 14, 2023

jackye1995 commented Jan 17, 2023

findinpath Jan 18, 2023

Choose a reason for hiding this comment

jackye1995 Jan 18, 2023

Choose a reason for hiding this comment

alexjo2144 commented Jan 18, 2023

alexjo2144 left a comment

Choose a reason for hiding this comment

alexjo2144 Jan 18, 2023

Choose a reason for hiding this comment

jackye1995 Jan 18, 2023

Choose a reason for hiding this comment

github-actions bot commented Feb 20, 2023

bitsondatadev commented Feb 21, 2023

jackye1995 commented Feb 23, 2023

bitsondatadev commented Feb 23, 2023 • edited Loading

electrum commented Feb 24, 2023

jackye1995 commented Feb 24, 2023

ebyhr commented Mar 8, 2023

jackye1995 commented Mar 9, 2023

bitsondatadev commented Mar 9, 2023 • edited Loading

bitsondatadev commented Mar 9, 2023

ebyhr commented Mar 9, 2023

bitsondatadev commented Mar 9, 2023

jackye1995 commented Mar 9, 2023

electrum commented Mar 9, 2023

bitsondatadev commented Mar 12, 2023

electrum commented Mar 14, 2023

findepi commented Mar 16, 2023

bitsondatadev commented Feb 23, 2023 •

edited

Loading

bitsondatadev commented Mar 9, 2023 •

edited

Loading