[Segment Replication] Benchmarking Segment Replication.

[mch2]

We would also like to use OpenSearch-Benchmark with a set specific to segment replication. We would like to be able to leverage this to test segrep with various cluster configurations.

Some high level questions we'd like to answer with benchmarking:

• What needs to be updated to measure refresh times with segrep?
• How does adjusting max segment size impact perf?
• What is the perf impact of adding additional replicas? At what replica count does the network become a bottleneck?
• What is the refresh time compared to docrep?
• Are recovery times impacted?

At a minimum we will need the following new metrics for replications:

• Total size in bytes
• Timing of total replication event.
• Timing of only file copy event.
• Replication lag, how long it takes for documents to become searchable on all replicas?

[dreamer-89]

@Poojita-Raj @kartg : Do you mind creating a separate issue(s) to track pending work on benchmarking. As there are few fixes already went in as part of 2.3 release, we can create new issue for 2.4 (or next version) releases.