Apply the dereference pushdown at the physical level on parquet in Iceberg

[findinpath]

Description

Fixes Parquet dereference pushdown on the physical level on #17156

ORC still open

Implementation overview for dereference pushdown at physical level in Parquet

PoC is the implementation for Hive

trino/plugin/trino-hive/src/main/java/io/trino/plugin/hive/parquet/ParquetPageSourceFactory.java

Lines 316 to 330 in adef5f4

	public static Optional<MessageType> getParquetMessageType(List<HiveColumnHandle> columns, boolean useColumnNames, MessageType fileSchema)
	{
	Optional<MessageType> message = projectSufficientColumns(columns)
	.map(projection -> projection.get().stream()
	.map(HiveColumnHandle.class::cast)
	.collect(toUnmodifiableList()))
	.orElse(columns).stream()
	.filter(column -> column.getColumnType() == REGULAR)
	.map(column -> getColumnType(column, fileSchema, useColumnNames))
	.filter(Optional::isPresent)
	.map(Optional::get)
	.map(type -> new MessageType(fileSchema.getName(), type))
	.reduce(MessageType::union);
	return message;
	}

What happens in the code:

• only sufficient columns (if doing SELECT parent.child, child only parent field will actually be selected) will be selected
• for each column, only the necessary information will be selected from the file schema sent to parquet reader

If nested.nested1level1.field4 gets selected and the column type hierarchy looks like this:

    nested
            field 1
            field 2
            nested1level1
                      field3
                      field4
            nested2level1
                      field5
                      field6 

for creating the paquet schema, there will be used the parquet type having the following hierarchy:

    nested
            nested1level1
                      field4

• the types are all unioned together to create a parquet Message that corresponds to only to the nested columns which are actually selected from the nested rows.

Additional context and related issues

Corresponding tests which check also that the dereference pushdown is effective on the storage layer exist already in BaseConnectorTest. See 6a4e483e#diff-6be05909e810c0224c1951c4102cad6256e9e088feb91e4c260756bfde89b6d5

Lookup for usage of

trino/testing/trino-testing/src/main/java/io/trino/testing/TestingConnectorBehavior.java

Line 69 in adef5f4

SUPPORTS_DEREFERENCE_PUSHDOWN(SUPPORTS_ROW_TYPE),

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# Iceberg
* Improve performance of reading ROW types from parquet files. ({issue}`17387`)

[ebyhr]

What's the difference from #17133?

[findinpath]

What's the difference from #17133?

@ebyhr this PR provides a way to dramatically reduce the amount of Parquet data read from the physical storage while selecting sub-fields from nested fields. See #17145 for details.

#17133 is another very useful improvement for being able to push down filters on Parquet so that entire row group get skipped for reading. cc @leetcode-1533

[findinpath]

Rebasing on master to ensure that the changes still work as expected.

[raunaqmorarka]

lgtm % minor comments