Move test to `AbstractTestHiveFileSystem` #18172

findinpath · 2023-07-07T12:47:16Z

Description

The test testRecursiveDirectoriesWithSpecialCharacters() is being
moved to AbstractTestHiveFileSystem for being able to test it on
multiple object storage file systems.

Contains cherry-pick from #18167

Fixes #18169

Additional context and related issues

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text:

# Section
* Fix some things. ({issue}`issuenumber`)

plugin/trino-hive/src/test/java/io/trino/plugin/hive/AbstractTestHiveFileSystem.java

plugin/trino-hive-hadoop2/src/test/java/io/trino/plugin/hive/AbstractTestHiveFileSystemS3.java

pajaks · 2023-07-14T14:12:02Z

plugin/trino-hive-hadoop2/src/test/java/io/trino/plugin/hive/AbstractTestHiveFileSystemS3.java

+                .map(Path::new)
+                .toList();
+        // Should not include any hidden files, folders, or nested files
+        assertEqualsIgnoreOrder(shallowListing, ImmutableList.of());


Maybe it would be good to have at least one non hidden file on this level to check if it's listed?

In such a case, the file would pop up in the listing.

Yes, the idea is to test listening on top level.

pajaks · 2023-07-14T14:37:44Z

plugin/trino-hive-hadoop2/src/test/java/io/trino/plugin/hive/AbstractTestHiveFileSystemS3.java

+        assertEqualsIgnoreOrder(shallowListing, ImmutableList.of());
+    }
+
+    protected void createDirectory(String bucketName, String key)


This function is almost the same as createFile. Can reduce this duplication?

I slightly modified createDirectory.
I find the duplication between them at the moment still small enough to warrant having two similar methods.

plugin/trino-hive/src/test/java/io/trino/plugin/hive/AbstractTestHiveFileSystem.java

plugin/trino-hive-hadoop2/src/test/java/io/trino/plugin/hive/AbstractTestHiveFileSystemS3.java

findinpath · 2023-07-18T13:13:21Z

Unrelated CI failure in ci / artifacts-check

Error:  io.trino.server.rpm.ServerIT.testUninstall[/home/runner/work/trino/trino/core/trino-server-rpm/target/trino-server-rpm-423-SNAPSHOT.noarch.rpm, 19](2)  Time elapsed: 301.798 s  <<< FAILURE!
org.testcontainers.containers.ContainerLaunchException: Container startup failed for image eclipse-temurin:19-jre-centos7

ebyhr · 2023-07-26T02:32:02Z

/test-with-secrets sha=d31d0fdded87d2396b0988a3e15d84894892e6f4

github-actions · 2023-07-26T02:34:59Z

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/5663986651

plugin/trino-hive-hadoop2/src/test/java/io/trino/plugin/hive/AbstractTestHiveFileSystemS3.java

The test `testRecursiveDirectoriesWithSpecialCharacters()` is being moved to `AbstractTestHiveFileSystem` for being able to test it on multiple object storage file systems.

findinpath · 2023-07-26T14:51:37Z

Failures: 
Error:    TestTrinoS3FileSystemAccessOperations.testSelectPartitionTable:142->AbstractTestQueryFramework.assertUpdate:405->AbstractTestQueryFramework.assertUpdate:410 » QueryFailed Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, com.amazonaws.auth.profile.ProfileCredentialsProvider@1bac8cb2: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@48b80fc7: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/]
Error:    TestTrinoS3FileSystemAccessOperations.testSelectPartitionTable:142->AbstractTestQueryFramework.assertUpdate:405->AbstractTestQueryFramework.assertUpdate:410 » QueryFailed Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, com.amazonaws.auth.profile.ProfileCredentialsProvider@1bac8cb2: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@48b80fc7: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/]
Error:    TestTrinoS3FileSystemAccessOperations.testSelectWithFilter:121->AbstractTestQueryFramework.assertUpdate:405->AbstractTestQueryFramework.assertUpdate:410 » QueryFailed Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, com.amazonaws.auth.profile.ProfileCredentialsProvider@1bac8cb2: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@48b80fc7: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/]
Error:    TestTrinoS3FileSystemAccessOperations.testSelectWithFilter:121->AbstractTestQueryFramework.assertUpdate:405->AbstractTestQueryFramework.assertUpdate:410 » QueryFailed Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, com.amazonaws.auth.profile.ProfileCredentialsProvider@1bac8cb2: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@48b80fc7: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/]

Failures unrelated to the current PR.
I can't reproduce them locally.

@ebyhr could you pls run again the PR with secrets?

ebyhr · 2023-07-26T21:47:13Z

/test-with-secrets sha=dc43f197824f4d8eb826903ea9c05987a3d6a2fc

github-actions · 2023-07-26T23:41:46Z

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/5674482714

findinpath · 2023-07-27T12:39:01Z

Unrelated failures in trino-iceberg and trino-iceberg (cloud-tests) in https://github.com/trinodb/trino/actions/runs/5674482714/job/15378926660

cc @ebyhr

cla-bot bot added the cla-signed label Jul 7, 2023

findinpath force-pushed the findinpath/test-hive-recursive-listing branch from 9093996 to eb4f75a Compare July 7, 2023 13:02

github-actions bot added tests:hive hive Hive connector labels Jul 7, 2023

findinpath self-assigned this Jul 7, 2023

findinpath requested review from findepi and pajaks July 10, 2023 08:27

findinpath force-pushed the findinpath/test-hive-recursive-listing branch from eb4f75a to 29762b5 Compare July 10, 2023 08:28

findepi approved these changes Jul 10, 2023

View reviewed changes

plugin/trino-hive/src/test/java/io/trino/plugin/hive/AbstractTestHiveFileSystem.java Show resolved Hide resolved

plugin/trino-hive/src/test/java/io/trino/plugin/hive/AbstractTestHiveFileSystem.java Show resolved Hide resolved

findinpath force-pushed the findinpath/test-hive-recursive-listing branch from 29762b5 to a0fbbaa Compare July 10, 2023 21:49

findinpath requested a review from findepi July 10, 2023 21:49

pajaks reviewed Jul 14, 2023

View reviewed changes

findinpath force-pushed the findinpath/test-hive-recursive-listing branch from 2c8ce54 to 32bb3ec Compare July 17, 2023 13:35

findinpath requested a review from pajaks July 17, 2023 13:35

pajaks approved these changes Jul 18, 2023

View reviewed changes

plugin/trino-hive-hadoop2/src/test/java/io/trino/plugin/hive/AbstractTestHiveFileSystemS3.java Outdated Show resolved Hide resolved

findinpath force-pushed the findinpath/test-hive-recursive-listing branch from 32bb3ec to d31d0fd Compare July 18, 2023 10:06

ebyhr approved these changes Jul 26, 2023

View reviewed changes

findinpath added 2 commits July 26, 2023 12:09

Move test to AbstractTestHiveFileSystem

a6cdc27

The test `testRecursiveDirectoriesWithSpecialCharacters()` is being moved to `AbstractTestHiveFileSystem` for being able to test it on multiple object storage file systems.

Test directory lister with hive paths with characters to be escaped

dc43f19

findinpath force-pushed the findinpath/test-hive-recursive-listing branch from d31d0fd to dc43f19 Compare July 26, 2023 10:18

ebyhr merged commit 2f513f6 into trinodb:master Jul 27, 2023

github-actions bot added this to the 423 milestone Jul 27, 2023

colebow mentioned this pull request Aug 2, 2023

Add Trino 423 release notes #18496

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move test to `AbstractTestHiveFileSystem` #18172

Move test to `AbstractTestHiveFileSystem` #18172

findinpath commented Jul 7, 2023 •

edited

Loading

pajaks Jul 14, 2023

findinpath Jul 17, 2023

pajaks Jul 18, 2023

pajaks Jul 14, 2023

findinpath Jul 17, 2023

findinpath commented Jul 18, 2023

ebyhr commented Jul 26, 2023

github-actions bot commented Jul 26, 2023 •

edited

Loading

findinpath commented Jul 26, 2023

ebyhr commented Jul 26, 2023

github-actions bot commented Jul 26, 2023 •

edited

Loading

findinpath commented Jul 27, 2023

Move test to AbstractTestHiveFileSystem #18172

Move test to AbstractTestHiveFileSystem #18172

Conversation

findinpath commented Jul 7, 2023 • edited Loading

Description

Additional context and related issues

Release notes

pajaks Jul 14, 2023

Choose a reason for hiding this comment

findinpath Jul 17, 2023

Choose a reason for hiding this comment

pajaks Jul 18, 2023

Choose a reason for hiding this comment

pajaks Jul 14, 2023

Choose a reason for hiding this comment

findinpath Jul 17, 2023

Choose a reason for hiding this comment

findinpath commented Jul 18, 2023

ebyhr commented Jul 26, 2023

github-actions bot commented Jul 26, 2023 • edited Loading

findinpath commented Jul 26, 2023

ebyhr commented Jul 26, 2023

github-actions bot commented Jul 26, 2023 • edited Loading

findinpath commented Jul 27, 2023

Move test to `AbstractTestHiveFileSystem` #18172

Move test to `AbstractTestHiveFileSystem` #18172

findinpath commented Jul 7, 2023 •

edited

Loading

github-actions bot commented Jul 26, 2023 •

edited

Loading

github-actions bot commented Jul 26, 2023 •

edited

Loading