Add hidden $path column in Iceberg

[osscm]

Description

Is this change a fix, improvement, new feature, refactoring, or other?

improvement

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

How would you describe this change to a non-technical end user or system administrator?

With this change , it will be possible to select the file path of the data file from the table. $path will be a hidden column.
Other connector's like Hive and DeltaLake are supporting it.
This is required when user wants to debug the data related issues, and can map the data file as well.

select col1, "$path" from foo

Related issues, pull requests, and links

• Fixes Add $path hidden column in Iceberg connector #8769

Documentation

( ) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

( ) No release notes entries required.
(x) Release notes entries required with the following suggested text:

# Iceberg
* Add support for hidden `$path` columns. ({issue}`11661`)

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Manish Malhotra.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Manish Malhotra.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Manish Malhotra.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[findinpath]

@osscm please do squash the commits.

[findinpath]

I think that number comes from here actually: https://iceberg.apache.org/spec/#reserved-field-ids

Thanks @alexjo2144 for the heads up.

Here is the source code reference for reserved metadata columns:

https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/MetadataColumns.java

@alexjo2144 there are already a few reserved column ids. What should we do eventually for dealing with further metadata columns (e.g. : $file_size, $file_modified_time) ? Can we somehow avoiding dealing with field ids for Iceberg metadata columns in Trino?

[osscm]

I think that number comes from here actually: https://iceberg.apache.org/spec/#reserved-field-ids

Thanks @alexjo2144 for the heads up.

Here is the source code reference for reserved metadata columns:

https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/MetadataColumns.java

@alexjo2144 there are already a few reserved column ids. What should we do eventually for dealing with further metadata columns (e.g. : $file_size, $file_modified_time) ? Can we somehow avoiding dealing with field ids for Iceberg metadata columns in Trino?

I think that number comes from here actually: https://iceberg.apache.org/spec/#reserved-field-ids

yes @alexjo2144 @findinpath I took the reference from https://github.com/apache/iceberg/blob/827b6c86108eec7b6de25f61022c7f8b5dda481c/core/src/main/java/org/apache/iceberg/MetadataColumns.java#L34-L42 (thanks to @RussellSpitzer for the discussion)

So, for the other hidden columns, we can go further Integer.Max_VALUE - 1
like $file_size = Integer.Max_VALUE - 2 and $file_modified_time = Integer.Max_VALUE - 3

[findinpath]

So, for the other hidden columns, we can go further Integer.Max_VALUE - 1
like $file_size = Integer.Max_VALUE - 2 and $file_modified_time = Integer.Max_VALUE - 3

In case we'll need extra columns, I think it would probably be a good idea to add them first to iceberg project to avoid eventual clashes with reserved columns which may appear in the next Iceberg releases.

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Manish Malhotra.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Manish Malhotra.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Manish Malhotra.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[mosabua]

Reviewed docs content only. That part looks good to me ;-)

[osscm]

thanks @mosabua for quick review!

[osscm]

Thanks a lot @findinpath, @alexjo2144 , @ebyhr, @mosabua for all the help and reviews!

[osscm]

Thanks @findepi for all the help!
Initiating the reviews and spending time!

Is it possible to merge this PR now? thanks

[ebyhr]

Merged, thanks!

[osscm]

Merged, thanks!

thanks a lot!