Add coercion test for unpartitioned tables in hive #16119

Praveen2112 · 2023-02-15T07:14:59Z

Description

Adds additional test coverage for coercion on un-partitioned tables in hive.

Release notes

(x) This is not user-visible or docs only and no release notes are required.

findepi · 2023-03-06T14:00:50Z

@ebyhr does this relate to yours #15938 ?

Praveen2112 · 2023-03-06T14:17:07Z

It kind of add partial test coverage to #15938 - only for ORC as of now.

testing/trino-product-tests/src/main/java/io/trino/tests/product/hive/TestHiveCoercion.java

testing/trino-product-tests/src/main/java/io/trino/tests/product/hive/BaseTestHiveCoercion.java

findepi · 2023-03-07T10:53:56Z

testing/trino-product-tests/src/main/java/io/trino/tests/product/hive/BaseTestHiveCoercion.java

+        if (expectedExceptionsWithTrinoContext().isEmpty()) {
+            assertEquals(ImmutableSet.copyOf(prestoReadColumns), expectedPrestoResults.keySet());


why assertion became conditional?

In case of unpartitioned tables we might not support all the column coercion as of now so we avoid the assertion when Trino throws some sort of exception.

add a code comment then

findepi · 2023-03-07T10:54:40Z

...ct-tests/src/main/java/io/trino/tests/product/hive/TestHiveCoercionOnUnPartitionedTable.java

+import static java.lang.String.format;
+import static java.util.Locale.ENGLISH;
+
+public class TestHiveCoercionOnUnPartitionedTable


UnPartitioned is one word

findepi · 2023-03-07T10:56:11Z

...ct-tests/src/main/java/io/trino/tests/product/hive/TestHiveCoercionOnUnPartitionedTable.java

+        }
+    }
+
+    @Requires(OrcRequirements.class)


I don't think Requirements Providers gives us any benefit, especially for mutable tables which aren't reused.
Plain old direct create table in a test would be better.

Maybe we can remove the Requirements Provider as a follow up as it would be common for both partitioned and unpartitioned tables.

...ct-tests/src/main/java/io/trino/tests/product/hive/TestHiveCoercionOnUnPartitionedTable.java

findepi · 2023-03-07T11:03:21Z

testing/trino-product-tests/src/main/java/io/trino/tests/product/hive/BaseTestHiveCoercion.java

        Set<String> unsupportedColumns = expectedExceptions.keySet().stream()
-                .filter(context -> context.hiveVersion().orElseThrow().equals(hiveVersion) && tableName.contains(context.format()))


why removed?

For Trino side failures we don't need check on hive version right ?

unless the failure is related to what hive version did modify the version

anyway, why are you changing this line to first add .orElseThrow() and then to remove the condition if all you need is remove the condition?

With the latest change by squashing the commits - this changeset got disappeared.

- varchar is also a primitive - tailing/trailing

Praveen2112 · 2023-03-07T12:50:11Z

@findepi AC

findepi · 2023-03-07T16:26:33Z

@Praveen2112 didn't re-review, but i see the build is failin

skrzypo987

lgtm % findepi's comments

findepi · 2023-03-09T21:55:51Z

testing/trino-product-tests/src/main/java/io/trino/tests/product/hive/TestHiveCoercion.java

@@ -881,11 +881,11 @@ private static List<List<?>> extract(List<TrinoArray> arrays)

    public static class ColumnContext
    {
-        private final String hiveVersion;
+        private final Optional<String> hiveVersion;


Make version optional in ColumnContext

can this commit include some description on why?
(sorry, i failed to infer this myself)

findepi · 2023-03-09T21:57:08Z

testing/trino-product-tests/src/main/java/io/trino/tests/product/hive/BaseTestHiveCoercion.java

+        if (expectedExceptionsWithTrinoContext().isEmpty()) {
+            assertEquals(ImmutableSet.copyOf(prestoReadColumns), expectedPrestoResults.keySet());


add a code comment then

findepi · 2023-03-09T21:58:17Z

testing/trino-product-tests/src/main/java/io/trino/tests/product/hive/BaseTestHiveCoercion.java

        Set<String> unsupportedColumns = expectedExceptions.keySet().stream()
-                .filter(context -> context.hiveVersion().orElseThrow().equals(hiveVersion) && tableName.contains(context.format()))


unless the failure is related to what hive version did modify the version

anyway, why are you changing this line to first add .orElseThrow() and then to remove the condition if all you need is remove the condition?

findepi · 2023-03-09T21:59:10Z

...ct-tests/src/main/java/io/trino/tests/product/hive/TestHiveCoercionOnUnpartitionedTable.java

+                       STORED AS
+                        """ + fileFormat);


redundant line break between STORED AS and ORC

It allows us to re-use `ColumnContext` in scenarios which are independent of hive version.

And rename TestHiveCoercion to TestHiveCoercionOnPartitionedTable

Praveen2112 · 2023-03-10T07:00:23Z

@findepi AC

findepi · 2023-03-10T11:07:57Z

LGTM

cla-bot bot added the cla-signed label Feb 15, 2023

github-actions bot added the tests:hive label Feb 15, 2023

Praveen2112 force-pushed the praveen/coercion_test_un_partitioned_table branch 2 times, most recently from 7724041 to 2ff3167 Compare February 23, 2023 13:00

Praveen2112 force-pushed the praveen/coercion_test_un_partitioned_table branch from 2ff3167 to f23b8bc Compare March 6, 2023 12:59

Praveen2112 marked this pull request as ready for review March 6, 2023 13:57

Praveen2112 requested review from ebyhr, findepi, phd3, kokosing, skrzypo987, huberty89 and atanasenko March 6, 2023 13:57

Praveen2112 force-pushed the praveen/coercion_test_un_partitioned_table branch 2 times, most recently from c6048db to 35b38d9 Compare March 7, 2023 07:39

Praveen2112 mentioned this pull request Mar 7, 2023

Support coercing char types of different lengths in hive #16402

Merged

findepi reviewed Mar 7, 2023

View reviewed changes

Minor fixes in comment

e1a8721

- varchar is also a primitive - tailing/trailing

Praveen2112 force-pushed the praveen/coercion_test_un_partitioned_table branch 2 times, most recently from aec09cc to fd20512 Compare March 7, 2023 12:48

Praveen2112 force-pushed the praveen/coercion_test_un_partitioned_table branch from fd20512 to e08a8dc Compare March 9, 2023 07:19

skrzypo987 reviewed Mar 9, 2023

View reviewed changes

Praveen2112 force-pushed the praveen/coercion_test_un_partitioned_table branch from e08a8dc to 3c863f5 Compare March 9, 2023 09:04

findepi approved these changes Mar 9, 2023

View reviewed changes

Praveen2112 added 2 commits March 10, 2023 12:29

Make version optional in ColumnContext

91845c2

It allows us to re-use `ColumnContext` in scenarios which are independent of hive version.

Use record for ColumnContext

cea2891

Praveen2112 added 2 commits March 10, 2023 12:29

Extract BaseTestHiveCoercion from TestHiveCoercion

67af1f9

And rename TestHiveCoercion to TestHiveCoercionOnPartitionedTable

Add coercion test for unpartitioned orc tables in hive

f454737

Praveen2112 force-pushed the praveen/coercion_test_un_partitioned_table branch from 3c863f5 to f454737 Compare March 10, 2023 06:59

Praveen2112 merged commit 4bfa7b6 into trinodb:master Mar 10, 2023

github-actions bot added this to the 411 milestone Mar 10, 2023

colebow mentioned this pull request Mar 14, 2023

Add Trino 411 release notes #16552

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add coercion test for unpartitioned tables in hive #16119

Add coercion test for unpartitioned tables in hive #16119

Praveen2112 commented Feb 15, 2023 •

edited

Loading

findepi commented Mar 6, 2023

Praveen2112 commented Mar 6, 2023

findepi Mar 7, 2023

Praveen2112 Mar 7, 2023

findepi Mar 9, 2023

findepi Mar 7, 2023

findepi Mar 7, 2023

Praveen2112 Mar 7, 2023

findepi Mar 7, 2023

Praveen2112 Mar 7, 2023

findepi Mar 9, 2023

Praveen2112 Mar 10, 2023

Praveen2112 commented Mar 7, 2023

findepi commented Mar 7, 2023

skrzypo987 left a comment

findepi Mar 9, 2023

findepi Mar 9, 2023

findepi Mar 9, 2023

findepi Mar 9, 2023

Praveen2112 commented Mar 10, 2023

findepi commented Mar 10, 2023

		if (expectedExceptionsWithTrinoContext().isEmpty()) {
		assertEquals(ImmutableSet.copyOf(prestoReadColumns), expectedPrestoResults.keySet());

		Set<String> unsupportedColumns = expectedExceptions.keySet().stream()
		.filter(context -> context.hiveVersion().orElseThrow().equals(hiveVersion) && tableName.contains(context.format()))

Add coercion test for unpartitioned tables in hive #16119

Add coercion test for unpartitioned tables in hive #16119

Conversation

Praveen2112 commented Feb 15, 2023 • edited Loading

Description

Release notes

findepi commented Mar 6, 2023

Praveen2112 commented Mar 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Praveen2112 commented Mar 7, 2023

findepi commented Mar 7, 2023

skrzypo987 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Praveen2112 commented Mar 10, 2023

findepi commented Mar 10, 2023

Praveen2112 commented Feb 15, 2023 •

edited

Loading