[Searchable Snapshots] Add support for RemoteShardsBalancer

[kotwanikunal]

Is your feature request related to a problem? Please describe.
The remote search shards for searchable snapshots will rely on the data cache which will be configured as a part of search nodes. The allocation process should take into account these remote shards while assigning, moving or balancing them across clusters such that they are only assigned to nodes with the specific search role.

Additionally, in order to incorporate the remote nature of shards, the balancing logic has to be updated for both the remote and local shards since the requirements in terms of disk, shard count and other variables that are used for weight based calculation will differ for these shards.

Describe the solution you'd like

• Implement a new RemoteShardsBalancer which will handle balancing and allocation for remote search shards within the cluster
• The new balancer should extend the ShardsBalancer interface introduced as a part of Refactor BalancedAllocator.Balancer to LocalShardsBalancer #4761

Describe alternatives you've considered

• Details: [Searchable Snapshot] [Design Proposal] Shard Allocator for Remote Search Shards #4759

Additional context

• [Searchable Snapshot] [Design Proposal] Shard Allocator for Remote Search Shards #4759

[DevPJ9]

Hello, @kotwanikunal I would like to contribute.

[kotwanikunal]

Hey @DevPJ9! Thanks for taking your time to contribute to OpenSearch.
I missed assigning myself yesterday but I have been working on this issue, and it's currently not possible for multiple contributors to work on this.

We would love to have you contribute. This is a great place to start contributions for OpenSearch.
Let me know if you need any help in getting started.

[DevPJ9]

Sure @kotwanikunal. I will look for some other issues. :)

[kotwanikunal]

Merged with #4870.