Do not cache Hadoop LocatedFileStatus objects

[sopel39]

LocatedFileStatus objects contain much more fields than are required by Trino. It doesn't make sense
to cache them.

Description

Non-technical explanation

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text:

# Section
* Fix some things. ({issue}`issuenumber`)

[sopel39]

Helps with #14313

[findepi]

Helps with #14313

Helps or fixes?

the io.trino.plugin.hive.fs.CachingDirectoryLister#cache currently is bounded by total number of files, but file information still varies (eg list of blocks). Also, it's harder to tune.
What about making it based on memory footprint?
i.e. let the user configure number of bytes the cache can take, and then let's weight entries by retained size?
like in

trino/plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TransactionLogAccess.java

Line 119 in 9859b91

.weigher((Weigher<String, DeltaLakeDataFileCacheEntry>) (key, value) -> Ints.saturatedCast(estimatedSizeOf(key) + value.getRetainedSizeInBytes()))

[findepi]

Do not cache Hadoop LocatedFileStatus objects

Let's maybe call it

Reduce CachingDirectoryLister memory footprint

Instead of caching Hadoop LocatedFileStatus objects which contain many 
fields we don't need, store only the information we need.

it;s OK to address memory-aware caching configuration (#14408 (comment)) as a follow-up

[sopel39]

it;s OK to address memory-aware caching configuration (#14408 (comment)) as a follow-up

I was actually thinking about reducing limits (if needed) and not too much complexity.

TransactionScopeCachingDirectoryLister is still unbounded (it's per query)

[findepi]

TransactionScopeCachingDirectoryLister is still unbounded (it's per query)

it's per transaction, and Hive connector supports transactions spanning multiple queries

a single table can perhaps comprise of a large number of partitions (even more so after @arhimondr 's #14225) and these of large number of files, so maybe it needs to be size limited?

For context, Delta connector has active files cache (getting list of files is more expensive there), and it has been proven to raise connector memory requirements significantly when that connector is in use.

[sopel39]

it's per transaction, and Hive connector supports transactions spanning multiple queries

I mean concurrent queries or long running queries (file listing will still be cached during transaction).

a single table can perhaps comprise of a large number of partitions (even more so after @arhimondr 's #14225) and these of large number of files, so maybe it needs to be size limited?

I'm not sure extra complexity of counting bytes is worth it compared to just dropping listing limit to 10_000 (although I'm not sure how effective cache is then).

Also alternative is having some global size limit for transactional cache (but one has to keep object lifecycle then).