Skip to content

Commit

Permalink
Showcase dereference pushdown limited effectiveness in Iceberg
Browse files Browse the repository at this point in the history
Dereference pushdown is effective on filtering
the data being read from the source so that only relevant
nested data is being provided by the connector to the engine.
However, the whole content corresponding to a nested column
is currently read from the data files.
  • Loading branch information
findinpath committed Apr 21, 2023
1 parent 0c99af4 commit 23a3f2b
Showing 1 changed file with 6 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -4941,11 +4941,16 @@ public void testProjectionPushdownReadsLessData()
getSession(),
selectQuery,
statsWithPushdown -> {
DataSize physicalDataSizeWithPushdown = statsWithPushdown.getPhysicalInputDataSize();
DataSize processedDataSizeWithPushdown = statsWithPushdown.getProcessedInputDataSize();
assertQueryStats(
sessionWithoutPushdown,
selectQuery,
statsWithoutPushdown -> assertThat(statsWithoutPushdown.getProcessedInputDataSize()).isGreaterThan(processedDataSizeWithPushdown),
statsWithoutPushdown -> {
//TODO (https://github.com/trinodb/trino/issues/17156) add dereference pushdown on the physical layer
assertThat(statsWithoutPushdown.getPhysicalInputDataSize()).isEqualTo(physicalDataSizeWithPushdown);
assertThat(statsWithoutPushdown.getProcessedInputDataSize()).isGreaterThan(processedDataSizeWithPushdown);
},
results -> assertEquals(results.getOnlyColumnAsSet(), expected));
},
results -> assertEquals(results.getOnlyColumnAsSet(), expected));
Expand Down

0 comments on commit 23a3f2b

Please sign in to comment.