[BUG] stop replication fails to cleanly stop in case of very small indices

[gbbafna]

Describe the bug

there is a race condition where in , the index is left with write block even after replication is stopped

example logs

followCluster-1/logs/followCluster.log:[2021-09-16T13:57:22,262][INFO ][o.o.r.r.RemoteClusterRepository] [followCluster-1] Restore successful for [leader_index_yrwsia][0]


followCluster-1/logs/opensearch.stdout.log:[2021-09-16T13:57:23,514][INFO ][o.o.r.a.s.TransportStopIndexReplicationAction] [followCluster-1] Index[leader_index_yrwsia] is in restoring stage


followCluster-1/logs/followCluster.log:[2021-09-16T13:57:25,515][INFO ][o.o.r.t.i.IndexReplicationTask] [followCluster-1] [leader_index_yrwsia] Adding index block for replication

followCluster-1/logs/followCluster.log:[2021-09-16T13:57:25,514][INFO ][o.o.r.t.i.IndexReplicationTask] [followCluster-1] [leader_index_yrwsia] Starting shard tasks
followCluster-1/logs/followCluster.log:[2021-09-16T13:57:25,515][INFO ][o.o.r.t.i.IndexReplicationTask] [followCluster-1] [leader_index_yrwsia] Adding index block for replication
followCluster-1/logs/followCluster.log:[2021-09-16T13:57:27,512][INFO ][o.o.r.t.i.IndexReplicationTask] [followCluster-1] [leader_index_yrwsia] Cancelling index replication stop

Expected behavior

We should move all the logic from TransportStopReplication to IndexTask . This would take care of all such race conditions.