[Segment Replication] Send Remote shard failure for lagging replicas

[mch2]

As last part of #4478, we need a mechanism to recognize shards are stale (with metrics introduced with #6520) for too long and send a remote shard failure. This is a remote failure as these metrics are collected by the primary shard.

I am thinking we add a background task that checks on a short schedule (30s) and invokes

OpenSearch/server/src/main/java/org/opensearch/cluster/action/shard/ShardStateAction.java

Line 247 in da244c2

public void remoteShardFailed(