Add more product tests coverage for native filesystem GCS and Azure #21958

anusudarsan · 2024-05-13T21:06:40Z

Release notes

(x) This is not user-visible or is docs only, and no release notes are required.

ebyhr · 2024-05-13T22:10:01Z

/test-with-secrets sha=c7a2a6a227c9d6e03270146169028a3b49a1c57b

github-actions · 2024-05-13T22:27:23Z

The CI workflow run with tests that require additional secrets has been started: https://github.com/trinodb/trino/actions/runs/9070514362

nineinchnick · 2024-05-14T13:31:05Z

testing/trino-product-tests/src/main/java/io/trino/tests/product/hive/TestHiveAzure.java

+    }
+
+    @Test(groups = {HIVE_AZURE, PROFILE_SPECIFIC_TESTS})
+    public void testBasicWriteOperations()


Would it make sense to add a test for reading external tables, and/or tables created by Hive? This would verify the compatibility of native FS libs with those other systems.

TestAzureBlobFileSystem tests this. but the GCS counterpart might be missing. I will add it

testing/trino-product-tests/src/main/java/io/trino/tests/product/iceberg/TestIcebergAzure.java

findinpath · 2024-05-14T15:46:42Z

testing/trino-product-tests/src/main/java/io/trino/tests/product/iceberg/TestIcebergAzure.java

+
+    @Test(groups = {ICEBERG_AZURE, PROFILE_SPECIFIC_TESTS})
+    public void testBasicWriteOperations()
+    {


I see that the tests present here are in line with io.trino.tests.product.deltalake.TestDeltaLakeGcs which is not really a compatibility test against Spark.
Ideally we should ensure that spark can read what trino writes and the other way around for all the storages.
Yes, this would involve potentially creating new testing docker images

ok this part wasnt clear to me. I will work on adding these

cc @ebyhr for visibility on the testing docker images front.

I will add spark compatibility tests as a follow up. addressed other comments

ebyhr · 2024-05-20T03:58:38Z

/test-with-secrets sha=fc3ae0584a8f19288711f8925da0a568cdbc5f1a

github-actions · 2024-05-20T04:00:27Z

The CI workflow run with tests that require additional secrets has been started: https://github.com/trinodb/trino/actions/runs/9153277764

testing/trino-product-tests/src/main/java/io/trino/tests/product/BaseTestTableFormats.java

findinpath · 2024-05-21T20:36:45Z

.../docker/presto-product-tests/conf/environment/multinode-gcs/core-site-overrides-template.xml

@@ -5,6 +5,11 @@
        <value>false</value>
    </property>

+    <property>


What happened when this parameter was not specified?

the with space test in Hive GCS failed. Caused by: io.trino.spi.TrinoException: java.lang.IllegalArgumentException: Invalid bucket name (....) or object name (env_multinode_gcs_..../test_path_special_character7ehvy1chkk/part=with space) This is true for legacy fs tests as well.

findinpath

LGTM % comment in special character handling in the partition name test.

anusudarsan · 2024-05-22T20:22:29Z

I added a new commit for spark compatibility tests 1828858 ptal. the image is based off from trinodb/docker-images#198

anusudarsan · 2024-05-24T18:43:14Z

@ebyhr can you re-trigger ci with secrets please?

ebyhr · 2024-05-24T21:28:27Z

/test-with-secrets sha=faa0d123b2a6011135ba585bc635b4bb79a33e9f

github-actions · 2024-05-24T21:29:32Z

The CI workflow run with tests that require additional secrets has been started: https://github.com/trinodb/trino/actions/runs/9230141387

testing/trino-product-tests/src/main/java/io/trino/tests/product/BaseTestTableFormats.java

This is testing native filesystem implementation

Also with native filesystem enabled

ebyhr · 2024-05-28T10:38:01Z

testing/trino-product-tests/src/main/java/io/trino/tests/product/BaseTestTableFormats.java

+        String sparkTableName = format("%s.test_compat.%s", getSparkCatalog(), baseTableName);
+        String trinoTableName = format("%s.test_compat.%s", getCatalogName(), baseTableName);
+        try {
+            onTrino().executeQuery(format("CREATE SCHEMA %s.test_compat WITH (location = '%s')", getCatalogName(), schemaLocation));


No need to handle in this PR, but we should avoid creating a new schema per test.

ebyhr · 2024-05-28T10:38:29Z

/test-with-secrets sha=dae880a9b3ae965391c7fed516bf105fd62e6b3e

github-actions · 2024-05-28T11:10:09Z

The CI workflow run with tests that require additional secrets has been started: https://github.com/trinodb/trino/actions/runs/9268029862

cla-bot bot added the cla-signed label May 13, 2024

anusudarsan requested review from ebyhr, findinpath and wendigo May 13, 2024 21:06

findinpath requested review from nineinchnick May 14, 2024 12:23

nineinchnick approved these changes May 14, 2024

View reviewed changes

findinpath reviewed May 14, 2024

View reviewed changes

testing/trino-product-tests/src/main/java/io/trino/tests/product/iceberg/TestIcebergAzure.java Show resolved Hide resolved

findinpath reviewed May 14, 2024

View reviewed changes

testing/trino-product-tests/src/main/java/io/trino/tests/product/iceberg/TestIcebergAzure.java Show resolved Hide resolved

findinpath reviewed May 14, 2024

View reviewed changes

anusudarsan force-pushed the anu/extend-product-tests-for-native-fs branch from c7a2a6a to fc3ae05 Compare May 17, 2024 22:00

findinpath reviewed May 21, 2024

View reviewed changes

testing/trino-product-tests/src/main/java/io/trino/tests/product/BaseTestTableFormats.java Outdated Show resolved Hide resolved

findinpath reviewed May 21, 2024

View reviewed changes

testing/trino-product-tests/src/main/java/io/trino/tests/product/BaseTestTableFormats.java Outdated Show resolved Hide resolved

findinpath reviewed May 21, 2024

View reviewed changes

anusudarsan force-pushed the anu/extend-product-tests-for-native-fs branch from fc3ae05 to e39e770 Compare May 22, 2024 15:39

anusudarsan mentioned this pull request May 22, 2024

Modify spark image to include azure and gcs dependency jars trinodb/docker-images#198

Merged

anusudarsan force-pushed the anu/extend-product-tests-for-native-fs branch from 1828858 to 72397e2 Compare May 23, 2024 18:18

anusudarsan added 2 commits May 24, 2024 12:40

Abstract logic for table formats testing in GCS product tests

e8de846

Add iceberg in gcs product tests

81fc5cd

anusudarsan force-pushed the anu/extend-product-tests-for-native-fs branch from 17f4c57 to faa0d12 Compare May 24, 2024 16:40

ebyhr approved these changes May 27, 2024

View reviewed changes

anusudarsan added 4 commits May 28, 2024 19:35

Extend azure product tests for various table formats

bb737df

Add hive product test for gcs filesystem

f93aefd

Add tests for table formats and special chars in location

0a542fe

This is testing native filesystem implementation

Add spark compatibility tests for GCS and Azure

dae880a

Also with native filesystem enabled

ebyhr force-pushed the anu/extend-product-tests-for-native-fs branch from faa0d12 to dae880a Compare May 28, 2024 10:36

ebyhr reviewed May 28, 2024

View reviewed changes

ebyhr merged commit eca7ab3 into trinodb:master May 28, 2024
46 checks passed

github-actions bot added this to the 449 milestone May 28, 2024

anusudarsan deleted the anu/extend-product-tests-for-native-fs branch May 28, 2024 13:02

colebow mentioned this pull request May 29, 2024

Add Trino 449 release notes #22195

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add more product tests coverage for native filesystem GCS and Azure #21958

Add more product tests coverage for native filesystem GCS and Azure #21958

anusudarsan commented May 13, 2024 •

edited by ebyhr

Loading

ebyhr commented May 13, 2024

github-actions bot commented May 13, 2024

nineinchnick May 14, 2024

anusudarsan May 14, 2024

findinpath May 14, 2024

anusudarsan May 14, 2024

findinpath May 15, 2024

anusudarsan May 17, 2024

ebyhr commented May 20, 2024

github-actions bot commented May 20, 2024

findinpath May 21, 2024

anusudarsan May 21, 2024 •

edited

Loading

findinpath left a comment

anusudarsan commented May 22, 2024

anusudarsan commented May 24, 2024

ebyhr commented May 24, 2024

github-actions bot commented May 24, 2024

ebyhr May 28, 2024

ebyhr commented May 28, 2024

github-actions bot commented May 28, 2024

Add more product tests coverage for native filesystem GCS and Azure #21958

Add more product tests coverage for native filesystem GCS and Azure #21958

Conversation

anusudarsan commented May 13, 2024 • edited by ebyhr Loading

Release notes

ebyhr commented May 13, 2024

github-actions bot commented May 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ebyhr commented May 20, 2024

github-actions bot commented May 20, 2024

Choose a reason for hiding this comment

anusudarsan May 21, 2024 • edited Loading

Choose a reason for hiding this comment

findinpath left a comment

Choose a reason for hiding this comment

anusudarsan commented May 22, 2024

anusudarsan commented May 24, 2024

ebyhr commented May 24, 2024

github-actions bot commented May 24, 2024

Choose a reason for hiding this comment

ebyhr commented May 28, 2024

github-actions bot commented May 28, 2024

anusudarsan commented May 13, 2024 •

edited by ebyhr

Loading

anusudarsan May 21, 2024 •

edited

Loading