Skip to content

Commit

Permalink
Showcase dereference pushdown limited effectiveness in Iceberg
Browse files Browse the repository at this point in the history
Dereference pushdown is effective on filtering
the data being read from the source so that only relevant
nested data is being provided by the connector to the engine.
However, the whole content corresponding to a nested column
is currently read from the data files.
  • Loading branch information
findinpath authored and raunaqmorarka committed Apr 24, 2023
1 parent 21608c7 commit 40ab0b3
Showing 1 changed file with 6 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -4930,11 +4930,16 @@ public void testProjectionPushdownReadsLessData()
getSession(),
selectQuery,
statsWithPushdown -> {
DataSize physicalDataSizeWithPushdown = statsWithPushdown.getPhysicalInputDataSize();
DataSize processedDataSizeWithPushdown = statsWithPushdown.getProcessedInputDataSize();
assertQueryStats(
sessionWithoutPushdown,
selectQuery,
statsWithoutPushdown -> assertThat(statsWithoutPushdown.getProcessedInputDataSize()).isGreaterThan(processedDataSizeWithPushdown),
statsWithoutPushdown -> {
//TODO (https://github.com/trinodb/trino/issues/17156) add dereference pushdown on the physical layer
assertThat(statsWithoutPushdown.getPhysicalInputDataSize()).isEqualTo(physicalDataSizeWithPushdown);
assertThat(statsWithoutPushdown.getProcessedInputDataSize()).isGreaterThan(processedDataSizeWithPushdown);
},
results -> assertEquals(results.getOnlyColumnAsSet(), expected));
},
results -> assertEquals(results.getOnlyColumnAsSet(), expected));
Expand Down

0 comments on commit 40ab0b3

Please sign in to comment.