Implement prototype remote store directory/index input for search

[yigithub]

No description provided.

[neetikasinghal]

Issue description:
Searchable snapshots implemented a RemoteSnapshotDirectory that provides access to files that are physically represented as a snapshot in a repository (the specific repository implementation is provided via a storage plugin). This task is to create a similar remote search-focused Directory implementation for searching remote-backed indexes stored in a repository. The bulk of the logic for the Lucene abstraction that implements the on-demand fetching and file caching is implemented in the class OnDemandBlockIndexInput and will be reused here.

The remote-backed storage feature has implemented classes like RemoteSegmentStoreDirectory and RemoteIndexInput. There is some overlap as these classes must also understand the remote segment structure and metadata. Part of this prototype is to figure out how the pieces fit together, but particularly at the IndexInput level there will be distinct implementations as the remote search piece is built around on-demand partial file fetching, whereas the pieces used in the write path are designed to copy entire files to/and from the remote object store.

[neetikasinghal]

High-level Design

In the case of Searchable Snapshots, FileInfo class (part of BlobStoreIndexShardSnapshot class) contains the metadata for each segment/data file like file name, part size etc. The FileInfo object is used to make an object of BlobFetchRequest that is helpful in fetching/downloading the blocks of data from the remote store during the read path.
In case of Remote Search, we can leverage the UploadedSegmentMetadata class and use it to form the object of BlobFetchRequest similar to Searchable Snapshots.

In current flow of remote store upload, there is a listener registered for each refresh, that takes care of uploading/updating the metadata of each shard to the remote store. Current metadata is stored under a RemoteDirectory:

• Location - <IndexUUID>/<Shard ID>/segments/metadata/
• File Name - metadata__<Primary Term>__<Commit Generation>__<UUID>
• Content in the file for every segment file uploaded at each commit: <OriginalSegmentFilename>::<UploadedSegmentFilename>::<Checksum>
• Length of the file name

Create a new Directory - RemoteSearchDirectory and new IndexInput inheritor of OnDemandBlockIndexInput - OnDemandBlockSearchIndexInput

Initialization of the directory
The initialization of the Search based directory can be done at RemoteSearchDirectoryFactory , similar to the initialization of RemoteDirectory.

Upload/Update/Delete of metadata to the directory
Upload of metadata happens after every commit, update can happen after each refresh, at remote store.
The stale commit commit metadata files are deleted.

Content of metadata file
This contains UploadedSegmentMetadata of each of the segment file uploaded

Read/Loading the metadata file in the directory
OnDemandBlockSearchIndexInput class will be created to read the files in the directory newly created.

End to end flow for testing
create an index → remote backup → close the index → apply the remote search setting → open the index back → perform search on the index
Note: This testing setup will work only for the immutable indexes.

[neetikasinghal]

Draft PR: #7417