[META] Composite Directory Implementation for Writable Warm

[rayshrey]

Please describe the end goal of this project

Implement Composite Directory - a new directory implementation where data is backed in a remote store and not all data needs to be stored locally. This directory will behave as a local directory when complete files are present in disk, but can fall back to the on-demand fetch(can be extended to block level or non block level fetch) from the remote store when data is is not present locally.

We will also need to enhance the present FileCache implementation as the implementation of Composite Directory is heavily dependent on this FileCache

Supporting References

Composite Directory RFC - #12781

Issues

Basic Components:

• [Writable Warm] Design and implement Composite Directory and integrate with FileCache #12781
• [Writable Warm] Recovery/Replication flow for Composite Directory #13647
• [Writable Warm] File Pinning support and TTL based eviction for FileCache #13648
• Add hooks for hot/warm migration in Composite Directory
• Add stats in FileCache

Follow-up AIs from the above PRs:

• Optimize readInput for Block files of RemoteDirectory to avoid loading whole 8MB chunk in memory
• Explore and verify impact of temp files in directory as they are not tracked in FileCache
• Add new role for Writable Warm for FileCache initialization logic (instead of being dependent on Feature Flag)
• Refactor OnDemandSnapshotIndexInput and FileInfo to more generic names as they are now being used by Composite directory as well which is not Snapshot related.
• Optimize and make listAll() of RemoteSegmentStoredircetory more robust (fix NPE on empty metadata and improve complexity)

Backlogs:

• Use String as key of FileCache instead of Path to avoid having hard dependency of local directory being FSDirectory

Related component

Storage:Remote