Support Metrics mode when creating/writing Iceberg tables #9938

liqinrae · 2021-11-11T01:16:09Z

Added support for all metrics modes in orc format as mentioned in #9791

Default metrics mode is changed to truncate(16)
Allow for configuring default mode to NONE so that we can skip all metrics by default except the ones we want
Allows per column metrics mode configuration to support different modes: counts, none, truncate(length) and full

cla-bot · 2021-11-11T01:16:11Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: q-li.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email [email protected]
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cla-bot · 2021-11-11T15:54:35Z

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please submit the signed CLA to [email protected]. For more information, see https://github.com/trinodb/cla.

cla-bot · 2021-11-11T17:23:59Z

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please submit the signed CLA to [email protected]. For more information, see https://github.com/trinodb/cla.

cla-bot · 2021-11-11T17:36:14Z

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please submit the signed CLA to [email protected]. For more information, see https://github.com/trinodb/cla.

rdsr · 2021-11-15T17:14:25Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergOrcFileWriter.java

+            if (metricsMode != MetricsModes.Counts.get()) {
+                toIcebergMinMax(orcColumnStats, icebergField.type(), metricsMode).ifPresent(minMax -> {


Do we not handle metrics mode truncate?

@rdsr yes we do, we handle truncate in the IcebergMinMax

I see we added support for this below

jackye1995 · 2021-11-29T23:51:05Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

@@ -431,7 +432,8 @@ public ConnectorOutputTableHandle beginCreateTable(ConnectorSession session, Con
                getColumns(transaction.table().schema(), typeManager),
                transaction.table().location(),
                getFileFormat(transaction.table()),
-                transaction.table().properties());
+                transaction.table().properties(),
+                Collections.emptyMap());


the whole table property map is already passed in as a part of the writable table handle, why do we need to construct the metrics config as a separated field?

@jackye1995 yes you are right. There is no need to construct the metrics config since the table properties have already been passed to the writable table handle. I checked the code and we already have functions in place to retrieve metrics related properties so we can make this even simpler. Will add a fix to this, thanks!

jackye1995 · 2021-11-30T01:27:58Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergPageSink.java

@@ -128,6 +131,7 @@ public IcebergPageSink(
        this.jsonCodec = requireNonNull(jsonCodec, "jsonCodec is null");
        this.session = requireNonNull(session, "session is null");
        this.fileFormat = requireNonNull(fileFormat, "fileFormat is null");
+        this.metricsConfig = MetricsConfig.fromProperties(requireNonNull(storageProperties, "metricsConfig is null"));


error message does not match.

Btw, I think it makes more sense to pass in the MetricsConfig to the constructor instead of storageProperties, because file format is also derived from storageProperties and we are passing it in separately.

@jackye1995 Sorry you meant passing in the MetricsConfig to the pagesink or IcebergWritableTableHandle constructor?

The file format is derived from the property map and then passed to the writable table handle as a separate component, which is how we passed inMetricsConfig initially.

to the page sink. But I don't have a strong opinion on this, there are many duplicated information passed across the codebase as of today. We can just fix the error message first.

@liqinrae can you please check the Ci build results?

@findepi Yes seems like there is new changes from other PRs. Will merge them and fix the CI build.

findepi · 2021-11-30T16:12:40Z

@liqinrae can you please check the Ci build results?

jackye1995

looks good to me

findepi

Please squash your commits.

When doing so, try not to change the base commit:

git rebase -i $(git merge-base head master)

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergFileWriterFactory.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergPageSink.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergPageSinkProvider.java

findepi · 2021-12-01T09:23:17Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergOrcFileWriter.java

-                upperBoundsBuilder.put(icebergId, minMax.getMax());
-            });
+
+            if (metricsMode != MetricsModes.Counts.get()) {


MetricsMode is an interface, open for extensions. The fact that something isn't Counts doesn't mean it's Full.

Also, there is no javadoc indicating .get() returns same instance every time, so using .equals() would be more appropriate here.

this should be handled in a defensive manner:

if (metricsMode.equals(MetricsModes.None.get())) { // nothing to do } else if (metricsMode.equals(MetricsModes.Counts.get()) { .. collect counts } else if (metricsMode.equals( full )) { ... } else { throw new UnsupportedOperationException("Unsupported metrics mode: " + metricsMode); }

@findepi The conditional statement here handles the case where the metricsMode is truncate(length) or full. Both of these two cases need to store min/max except that truncate need to apply truncation if needed.

All of the metrics modes other than None need to collect counts. There would be duplication of code if we are to handle each mode separately. Would adding a verify function here more feasible if we want to handle it more defensively?

findepi · 2021-12-01T09:26:41Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergOrcFileWriter.java

-            this.max = Conversions.toByteBuffer(type, max);
+            // Out of the two types which could be truncated, string or binary, ORC only supports string bounds.
+            // Therefore, truncation will be applied if needed only on string type.
+            if (metricsMode instanceof MetricsModes.Truncate && type.typeId() == org.apache.iceberg.types.Type.TypeID.STRING) {


binary (varbinary) and fixed too

https://github.com/apache/iceberg/blob/2e4684740b56f6d80328f50b7e7fef0863c262e7/parquet/src/main/java/org/apache/iceberg/parquet/ParquetUtil.java#L269

(we don't support creating tables with fixed, but i think we do support writes to such tables)

@findepi I was referring to the OrcMetrics implementation here: https://github.com/apache/iceberg/blob/ddc5aff9167c7e367e2b4313b91e55451f09ef16/orc/src/main/java/org/apache/iceberg/orc/OrcMetrics.java#L281
Seems like it only added truncation for string bound in the orc format.

because they don't produce any stats for binary for ORC, see

trino/plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergOrcConnectorTest.java

Line 31 in 88ef2ca

return !(typeName.equalsIgnoreCase("varbinary"));

right?

still, not having an if for varbinary doesn't seem future proof, does it?

@findepi yes I agree. I'll add varbinary here.

findepi · 2021-12-01T09:27:57Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergOrcFileWriter.java

+                this.min = Conversions.toByteBuffer(type, min);
+                this.max = Conversions.toByteBuffer(type, max);


if MetricsMode is != Full (eg Counts, None), no min/max should be stored.

again, this should be handled in defensive manner, instead of if-ing on a Full class

findepi · 2021-12-01T09:29:34Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergOrcMetricsCollection.java

@@ -129,7 +249,7 @@ public void testBasic()
        assertQuery("SELECT min(orderpriority) FROM tpch.tiny.orders", "VALUES '" + lowerBounds.get(6) + "'");
        assertQuery("SELECT min(clerk) FROM tpch.tiny.orders", "VALUES '" + lowerBounds.get(7) + "'");
        assertQuery("SELECT min(shippriority) FROM tpch.tiny.orders", "VALUES " + lowerBounds.get(8));
-        assertQuery("SELECT min(comment) FROM tpch.tiny.orders", "VALUES '" + lowerBounds.get(9) + "'");
+        assertQuery("SELECT substring(min(comment), 1, 16) FROM tpch.tiny.orders", "VALUES '" + lowerBounds.get(9) + "'");


lowerBounds.get(9) is the actual, while SELECT substring(min(comment), 1, 16) FROM tpch.tiny.orders is the expected. The expected/actual are reversed here.

@findepi I tried reversing the order for all the assertQuery in the test function but encountered failures when using SELECT substring(min(comment), 1, 16) FROM tpch.tiny.orders as the expected sql statement. Checked other test files and looks like they placed all select statement as the expected and result as the actual.

we shouldn't use assertQuery at all, when we want to assert a value that we have at hand.

this is preexisting, so ok to followup

liqinrae · 2021-12-09T01:04:34Z

Hi @findepi @jackye1995 I addressed the comments and left a few replies on this PR. Please let me know your thoughts. Thanks!

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergFileWriterFactory.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergOrcFileWriter.java

findepi · 2022-01-14T13:09:34Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergOrcMetricsCollection.java

@@ -129,7 +249,7 @@ public void testBasic()
        assertQuery("SELECT min(orderpriority) FROM tpch.tiny.orders", "VALUES '" + lowerBounds.get(6) + "'");
        assertQuery("SELECT min(clerk) FROM tpch.tiny.orders", "VALUES '" + lowerBounds.get(7) + "'");
        assertQuery("SELECT min(shippriority) FROM tpch.tiny.orders", "VALUES " + lowerBounds.get(8));
-        assertQuery("SELECT min(comment) FROM tpch.tiny.orders", "VALUES '" + lowerBounds.get(9) + "'");
+        assertQuery("SELECT substring(min(comment), 1, 16) FROM tpch.tiny.orders", "VALUES '" + lowerBounds.get(9) + "'");


we shouldn't use assertQuery at all, when we want to assert a value that we have at hand.

this is preexisting, so ok to followup

findepi · 2022-01-14T13:10:44Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergOrcMetricsCollection.java

@@ -141,7 +261,8 @@ public void testBasic()
        assertQuery("SELECT max(orderpriority) FROM tpch.tiny.orders", "VALUES '" + upperBounds.get(6) + "'");
        assertQuery("SELECT max(clerk) FROM tpch.tiny.orders", "VALUES '" + upperBounds.get(7) + "'");
        assertQuery("SELECT max(shippriority) FROM tpch.tiny.orders", "VALUES " + upperBounds.get(8));
-        assertQuery("SELECT max(comment) FROM tpch.tiny.orders", "VALUES '" + upperBounds.get(9) + "'");
+        // the default truncate(16) mode would round up the last character of the maximum bound while keeping the first 15 characters identical
+        assertQuery("SELECT substring(max(comment), 1, 15) FROM tpch.tiny.orders", "VALUES '" + upperBounds.get(9).substring(0, 15) + "'");


assert on upperBounds.get(9) without taking substring

on the left side, use substring(max(comment), 1, 15) || '...' to construct the expected value

findepi · 2022-01-18T15:33:39Z

@liqinrae have you had time to update this for my comments?

liqinrae · 2022-01-18T22:46:19Z

@liqinrae have you had time to update this for my comments?

@findepi Hey thanks for reviewing my changes. Updated the PR for your comment and also replaced assertQuery in the preexisting test class with the actual min/max value. Please review and let me know your thoughts. Thanks.

rdsr reviewed Nov 15, 2021

View reviewed changes

cla-bot bot added the cla-signed label Nov 22, 2021

liqinrae requested review from findepi, jackye1995 and electrum November 29, 2021 23:47

jackye1995 reviewed Nov 29, 2021

View reviewed changes

jackye1995 reviewed Nov 30, 2021

View reviewed changes

liqinrae requested a review from jackye1995 November 30, 2021 21:10

jackye1995 approved these changes Dec 1, 2021

View reviewed changes

findepi reviewed Dec 1, 2021

View reviewed changes

liqinrae force-pushed the metrics_mode branch from 031c2fd to 000ad30 Compare December 3, 2021 00:09

liqinrae requested review from jackye1995 and findepi January 11, 2022 17:56

findepi approved these changes Jan 14, 2022

View reviewed changes

liqinrae force-pushed the metrics_mode branch from 000ad30 to 3c81be5 Compare January 18, 2022 22:42

liqinrae requested a review from findepi January 18, 2022 22:46

liqinrae force-pushed the metrics_mode branch 2 times, most recently from 7f52b71 to f319189 Compare January 19, 2022 06:34

customize metrics mode

ffb9007

liqinrae force-pushed the metrics_mode branch from 3040381 to ffb9007 Compare January 19, 2022 06:40

findepi merged commit 835438a into trinodb:master Jan 19, 2022

findepi mentioned this pull request Jan 19, 2022

Release notes for 369 #10552

Closed

mosabua mentioned this pull request Jan 20, 2022

Add Trino 369 release notes #10553

Merged

findepi mentioned this pull request Feb 8, 2022

Improve the performance of Min/Max/Count utilizing Iceberg table metrics #10974

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Metrics mode when creating/writing Iceberg tables #9938

Support Metrics mode when creating/writing Iceberg tables #9938

liqinrae commented Nov 11, 2021

cla-bot bot commented Nov 11, 2021

cla-bot bot commented Nov 11, 2021

cla-bot bot commented Nov 11, 2021

cla-bot bot commented Nov 11, 2021

rdsr Nov 15, 2021

liqinrae Nov 15, 2021

rdsr Nov 15, 2021

jackye1995 Nov 29, 2021

liqinrae Nov 30, 2021

jackye1995 Nov 30, 2021

liqinrae Nov 30, 2021

jackye1995 Nov 30, 2021

liqinrae Nov 30, 2021

findepi commented Nov 30, 2021

jackye1995 left a comment

findepi left a comment

findepi Dec 1, 2021

liqinrae Dec 2, 2021 •

edited

Loading

findepi Dec 1, 2021

liqinrae Dec 1, 2021

findepi Dec 1, 2021

findepi Dec 1, 2021

liqinrae Dec 1, 2021

findepi Dec 1, 2021

findepi Dec 1, 2021

liqinrae Dec 9, 2021

findepi Jan 14, 2022

liqinrae commented Dec 9, 2021

findepi Jan 14, 2022

findepi Jan 14, 2022

findepi commented Jan 18, 2022

liqinrae commented Jan 18, 2022

		if (metricsMode != MetricsModes.Counts.get()) {
		toIcebergMinMax(orcColumnStats, icebergField.type(), metricsMode).ifPresent(minMax -> {

		this.min = Conversions.toByteBuffer(type, min);
		this.max = Conversions.toByteBuffer(type, max);

Support Metrics mode when creating/writing Iceberg tables #9938

Support Metrics mode when creating/writing Iceberg tables #9938

Conversation

liqinrae commented Nov 11, 2021

cla-bot bot commented Nov 11, 2021

cla-bot bot commented Nov 11, 2021

cla-bot bot commented Nov 11, 2021

cla-bot bot commented Nov 11, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findepi commented Nov 30, 2021

jackye1995 left a comment

Choose a reason for hiding this comment

findepi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liqinrae Dec 2, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liqinrae commented Dec 9, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findepi commented Jan 18, 2022

liqinrae commented Jan 18, 2022

liqinrae Dec 2, 2021 •

edited

Loading