Migrate Hive DirectoryLister to TrinoFileSystem

[alexjo2144]

Description

Refactoring some of the Hadoop FileSystem uses to instead use TrinoFileSystem.

Additional context and related issues

Removing the hierarchical listing API from the DirectoryLister was a necessary requirement for the migration, since TrinoFileSystem does not have a directory based listing API, only a recursive one.

There are a couple functional changes that I think are unavoidable:
- Reading from Hive tables pointing to a location that does not exist produces no rows rather than failing
- hive.ignore-absent-partitions now applies both to partitions whose directories do not exist, but also ones whose directories exist but are empty
Edit: There are no longer any behavior changes

Release notes

(x) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text:

[alexjo2144]

Note to self, DirectoryListingCacheKey is unused now. Just didn't want to interrupt the build for that update.

[alexjo2144]

io.trino.tests.product.hive.TestAvroSymlinkInputFormat.testSymlinkTableWithNestedDirectory is the last failure I don't understand. It passes when I run it individually but fails when run as the full set like in suite-5 -g 'configured_features, storage_formats, hdfs_impersonation'

[electrum]

Why do we need "Support Locations without authorities on all schemes"?

[electrum]

Nit: put the file system argument first

Not sure about first commit

[alexjo2144]

Why do we need "Support Locations without authorities on all schemes"?

The product tests hit a few cases where we were listing locations like hdfs:/a/table/path without an authority/host present. This seems to line up with the Wikipedia entry for URIs which says the :// prefix notes that an authority is present, otherwise the scheme is directly followed by the path.

[alexjo2144]

Just a fixed checkstyle error

[electrum]

Not sure if this is related or flaky:

Error:  io.trino.hdfs.TestFileSystemCache.testFileSystemCacheConcurrency  Time elapsed: 0.825 s  <<< FAILURE!
java.lang.AssertionError: Cache size is non zero expected [0] but found [1]
	at org.testng.Assert.fail(Assert.java:94)
	at org.testng.Assert.failNotEquals(Assert.java:513)
	at org.testng.Assert.assertEqualsImpl(Assert.java:135)
	at org.testng.Assert.assertEquals(Assert.java:116)
	at org.testng.Assert.assertEquals(Assert.java:284)
	at io.trino.hdfs.TestFileSystemCache.testFileSystemCacheConcurrency(TestFileSystemCache.java:118)

[alexjo2144]

Comments applied, had to resolve some conflicts from #17432 as well

[alexjo2144]

And looks like TestFileSystemCache was just a flake: #17158

[electrum]

TestFileSystemCache failed again. Retrying it now. It seems suspicious, but this PR doesn't seem to touch anything that could affect it, so maybe we were just unlucky twice.

[yohengyang]

trino/lib/trino-hdfs/src/main/java/io/trino/filesystem/hdfs/HdfsFileSystem.java

Lines 199 to 222 in fad8847

	public Optional<Boolean> directoryExists(Location location)
	throws IOException
	{
	stats.getDirectoryExistsCalls().newCall();
	Path directory = hadoopPath(location);
	FileSystem fileSystem = environment.getFileSystem(context, directory);
	return environment.doAs(context.getIdentity(), () -> {
	if (!hierarchical(fileSystem, location)) {
	return Optional.empty();
	}
	try (TimeStat.BlockTimer ignored = stats.getDirectoryExistsCalls().time()) {
	FileStatus fileStatus = fileSystem.getFileStatus(directory);
	return Optional.of(fileStatus.isDirectory());
	}
	catch (FileNotFoundException e) {
	return Optional.of(false);
	}
	catch (IOException e) {
	stats.getListFilesCalls().recordException(e);
	throw e;
	}
	});

@alexjo2144 @electrum Is it necessary to use the hierarchical method in HdfsFileSystem to determine whether the file system is a Hierarchical file system or an Object store file system? Because when using HdfsFileSystem, it is certain that it is not an Object store file system.

[alexjo2144]

Is it necessary to use the hierarchical method in HdfsFileSystem

It is for now, because we haven't introduced native TrinoFileSystem implementations for all file systems yet. HdfsFileSystem is still used for s3/ Azure FSs, GCS, etc. Once it is only used to connect to HDFS we will be able to take it out.

[yohengyang]

HdfsFileSystem is still used for s3/ Azure FSs, GCS, etc. Once it is only used to connect to HDFS we will be able to take it out.

Tks for replying, got it