Update iceberg to 0.13.1 #11032

homar · 2022-02-14T11:20:09Z

Description

Updates apache iceberg version to 0.13.1

General information

Related issues, pull requests, and links

Documentation

(x) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

(x) No release notes entries required.
( ) Release notes entries required with the following suggested text:

# Section
* Fix some things. ({issue}`5678`)

findepi · 2022-02-14T11:43:33Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergOrcFileWriter.java

@@ -104,7 +104,8 @@ public Metrics getMetrics()
    private static Metrics computeMetrics(MetricsConfig metricsConfig, Schema icebergSchema, ColumnMetadata<OrcType> orcColumns, long fileRowCount, Optional<ColumnMetadata<ColumnStatistics>> columnStatistics)
    {
        if (columnStatistics.isEmpty()) {
-            return new Metrics(fileRowCount, null, null, null, null, null);
+            // TODO add implementation of Map<Integer, Long> nanValueCounts


Make it a Note, not TODO. CBO has no use for NaN counts today

findepi · 2022-02-14T11:44:58Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestMetricsWrapper.java

@@ -46,7 +46,7 @@ public void testRoundTrip()
        Map<Integer, ByteBuffer> lowerBounds = ImmutableMap.of(13, ByteBuffer.wrap(new byte[] {0, 8, 9}));
        Map<Integer, ByteBuffer> upperBounds = ImmutableMap.of(17, ByteBuffer.wrap(new byte[] {5, 4, 0}));

-        Metrics expected = new Metrics(recordCount, columnSizes, valueCounts, nullValueCounts, lowerBounds, upperBounds);
+        Metrics expected = new Metrics(recordCount, columnSizes, valueCounts, nullValueCounts, null, lowerBounds, upperBounds);


empty map instead?

findepi · 2022-02-14T11:45:54Z

@homar are you aware of any changes impacting users that flows from this upgrade?

homar · 2022-02-14T12:18:43Z

@homar are you aware of any changes impacting users that flows from this upgrade?

I am not. Main reason for this update is that it should allow us to implement row level deletes and updates for orc files.

electrum · 2022-02-16T00:11:38Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/MetricsWrapper.java

@@ -40,6 +41,7 @@ public MetricsWrapper(
                columnSizes,
                valueCounts,
                nullValueCounts,
+                ImmutableMap.of(), // Trino CBO doesn't utilise nanValueCounts at all


These are Iceberg metrics that are part of the Iceberg specification, and thus we should be writing them. The purpose of MetricsWrapper is to allow JSON serialization of Iceberg Metrics, so we should pass the value through here. The Parquet writer already collects NaN counts, and the ORC writer should be updated to collect them in the future.

Saying "Trino doesn't use them so we don't care" is hostile.

electrum · 2022-02-16T00:15:19Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergOrcFileWriter.java

@@ -104,7 +104,8 @@ public Metrics getMetrics()
    private static Metrics computeMetrics(MetricsConfig metricsConfig, Schema icebergSchema, ColumnMetadata<OrcType> orcColumns, long fileRowCount, Optional<ColumnMetadata<ColumnStatistics>> columnStatistics)
    {
        if (columnStatistics.isEmpty()) {
-            return new Metrics(fileRowCount, null, null, null, null, null);
+            // Trino CBO doesn't utilise nanValueCounts at all


Remove comment

electrum · 2022-02-16T00:15:58Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergOrcFileWriter.java

@@ -153,6 +154,7 @@ private static Metrics computeMetrics(MetricsConfig metricsConfig, Schema iceber
                null, // TODO: Add column size accounting to ORC column writers
                valueCounts.isEmpty() ? null : valueCounts,
                nullCounts.isEmpty() ? null : nullCounts,
+                null, // Trino CBO doesn't utilise nanValueCounts at all


Replace comment with

// TODO: Add nanValueCounts to ORC writer

homar · 2022-02-18T08:17:57Z

@electrum could you please take another look ?

alexjo2144 · 2022-02-22T18:47:48Z

One of the changes in 0.13 was to deprecate the write.folder-storage.path and write.object-storage.path properties, replacing them with write.data.path
apache/iceberg#3094

Can you add an appropriate test to TestIcebergSparkCompatibility similar to the existing one for the deprecated properties?
https://github.com/trinodb/trino/blob/master/testing/trino-product-tests/src/main/java/io/trino/tests/product/iceberg/TestIcebergSparkCompatibility.java#L822

electrum

Minor comments, otherwise looks good

electrum · 2022-02-23T16:45:58Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergOrcMetricsCollection.java

@@ -237,7 +237,7 @@ public void testBasic()

        // Check NaN value count
        // TODO: add more checks after NaN info is collected
-        assertNull(datafile.getNanValueCounts());
+        assertEquals(datafile.getNanValueCounts(), null);


electrum · 2022-02-23T16:46:05Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergOrcMetricsCollection.java

@@ -328,7 +328,7 @@ public void testWithNaNs()
        datafile.getValueCounts().values().forEach(valueCount -> assertEquals(valueCount, (Long) 3L));

        // TODO: add more checks after NaN info is collected
-        assertNull(datafile.getNanValueCounts());
+        assertEquals(datafile.getNanValueCounts(), null);


electrum · 2022-02-23T16:46:57Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestMetricsWrapper.java

@@ -46,7 +46,7 @@ public void testRoundTrip()
        Map<Integer, ByteBuffer> lowerBounds = ImmutableMap.of(13, ByteBuffer.wrap(new byte[] {0, 8, 9}));
        Map<Integer, ByteBuffer> upperBounds = ImmutableMap.of(17, ByteBuffer.wrap(new byte[] {5, 4, 0}));

-        Metrics expected = new Metrics(recordCount, columnSizes, valueCounts, nullValueCounts, lowerBounds, upperBounds);
+        Metrics expected = new Metrics(recordCount, columnSizes, valueCounts, nullValueCounts, ImmutableMap.of(), lowerBounds, upperBounds);


Add values so we test the serialization

We should also test NaN counts for Parquet

I don't see any tests for metrics for Parquet format. Nothing like TestIcebergOrcMetricsCollection exists for parquet. I will add a ticket to create such a test

findinpath · 2022-02-24T03:28:03Z

In case that this PR gets merged only after #11062 please do remove the Iceberg related code for deleting table metadata files in io.trino.plugin.iceberg.TrinoHiveCatalog#dropTable

cla-bot bot added the cla-signed label Feb 14, 2022

losipiuk approved these changes Feb 14, 2022

View reviewed changes

findepi approved these changes Feb 14, 2022

View reviewed changes

homar force-pushed the homar/update_iceberg branch 2 times, most recently from 60df744 to c8dcd1e Compare February 15, 2022 09:39

losipiuk approved these changes Feb 15, 2022

View reviewed changes

electrum requested changes Feb 16, 2022

View reviewed changes

homar changed the title ~~Update iceberg to 0.13.0~~ Update iceberg to 0.13.1 Feb 16, 2022

Reformat TrinoHiveCatalog

3b6e89c

homar force-pushed the homar/update_iceberg branch 2 times, most recently from 44e5b7c to 922ea65 Compare February 16, 2022 14:21

losipiuk approved these changes Feb 17, 2022

View reviewed changes

findinpath mentioned this pull request Feb 18, 2022

Delete Iceberg table data on DROP #11062

Merged

electrum approved these changes Feb 23, 2022

View reviewed changes

homar force-pushed the homar/update_iceberg branch from 922ea65 to 33685cd Compare February 24, 2022 06:54

Update iceberg to 0.13.1

d032b67

homar force-pushed the homar/update_iceberg branch from 33685cd to d032b67 Compare February 24, 2022 07:12

github-actions bot added the tests:hive label Feb 24, 2022

losipiuk merged commit 7e1c3b0 into trinodb:master Feb 24, 2022

github-actions bot added this to the 372 milestone Feb 24, 2022

mosabua mentioned this pull request Feb 24, 2022

Add Trino 372 release notes #11094

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update iceberg to 0.13.1 #11032

Update iceberg to 0.13.1 #11032

homar commented Feb 14, 2022 •

edited

Loading

findepi Feb 14, 2022

findepi Feb 14, 2022

findepi commented Feb 14, 2022

homar commented Feb 14, 2022

electrum Feb 16, 2022 •

edited

Loading

electrum Feb 16, 2022

electrum Feb 16, 2022

homar commented Feb 18, 2022

alexjo2144 commented Feb 22, 2022

electrum left a comment

electrum Feb 23, 2022

electrum Feb 23, 2022

electrum Feb 23, 2022

electrum Feb 23, 2022

homar Feb 24, 2022

homar Feb 24, 2022

findinpath commented Feb 24, 2022

Update iceberg to 0.13.1 #11032

Update iceberg to 0.13.1 #11032

Conversation

homar commented Feb 14, 2022 • edited Loading

Description

General information

Related issues, pull requests, and links

Documentation

Release notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findepi commented Feb 14, 2022

homar commented Feb 14, 2022

electrum Feb 16, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

homar commented Feb 18, 2022

alexjo2144 commented Feb 22, 2022

electrum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findinpath commented Feb 24, 2022

homar commented Feb 14, 2022 •

edited

Loading

electrum Feb 16, 2022 •

edited

Loading