[Segment Replication]. Support graceful replication failure.

[mch2]

Right now when there is any error on replicas during the replication process the shard breaks and is not pulled from rotation or recreated.

The shard should fail gracefully by being removed from allocation and recreated. When rebooted the shard should be able to recover and restart as new by picking up the next replication checkpoint.

[mch2]

Closing this as outdated. Replicas will fail & be removed from rotation when a failure occurs. https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/indices/replication/SegmentReplicationTargetService.java#L112