Reduce Hive file system listing #18179
Conversation
I was finding that cache hits were somewhat uncommon because TrinoFileSystem instances are fairly short lived in the Hive code.
No, unfortunately not. ADLS Gen2 can be used in either a hierarchical or non-hierarchical mode and I don't think it is indicated by a different scheme. |
alexjo2144
force-pushed
the
hive/reduce-list-calls
branch
2 times, most recently
from
40bbfd9 to
67a5c7b
Compare
alexjo2144
force-pushed
the
hive/reduce-list-calls
branch
from
67a5c7b to
7dcc4a4
Compare
Reduce the number of times Hive needs to call the file system listing API when using common file systems. Additionally, avoid unnecessary directory exists checks when file listing returns a non-empty result.
alexjo2144
force-pushed
the
hive/reduce-list-calls
branch
from
7dcc4a4 to
f0ef639
Compare
| try { | ||
| return ImmutableList.copyOf(new HiveFileIterator(table, location, fs, directoryLister, hdfsNamenodeStats, FAIL)); | ||
| HiveFileIterator fileIterator = new HiveFileIterator(table, location, fs, directoryLister, hdfsNamenodeStats, FAIL); |
There was a problem hiding this comment.
Will iterator creation report error if location does not exists? If so will this error be as nice as one generated by checkPartitionLocationExists?
There was a problem hiding this comment.
Will iterator creation report error if location does not exists?
It will not, that's why the check is there.
Description
Reduce the number of times Hive needs to call the file system listing API when using common file systems.
Additional context and related issues
Relates to: #17323
Release notes
(x) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text: