[BUG] Flaky test - testRestoreShardFromRemoteStore

[mch2]

  2> REPRODUCE WITH: gradlew ':server:test' --tests "org.opensearch.index.shard.IndexShardTests.testRestoreShardFromRemoteStore" -Dtests.seed=709E54060BABD98F -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sr-ME -Dtests.timezone=Europe/Kiev -Druntime.java=17
  2> java.lang.AssertionError: expected:<0> but was:<2>
        at __randomizedtesting.SeedInfo.seed([709E54060BABD98F:DAD18613CF69A4F0]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.opensearch.index.shard.IndexShardTests.testRestoreShardFromRemoteStore(IndexShardTests.java:2710)

Run on windows CI against 2.4.

[Bukhtawar]

cc: @sachinpkale

[mch2]

Another instance - https://build.ci.opensearch.org/job/gradle-check/7152/testReport/junit/org.opensearch.index.shard/IndexShardTests/testRestoreShardFromRemoteStore_2/.

Having trouble reproing locally.

[sachinpkale]

Let me take a look and fix it on priority.

[mch2]

Looks like a windows specific issue. I tried deleting in a loop while length > 0 and all files will get deleted. I think files are moving to pendingDeleteFiles and not getting purged until another check.

I've hit this with 1 file remaining. Usually write.lock and 2 files remaining write.lock and .cfe.

[mch2]

@sachinpkale Put up a PR to fix this, its not specific to remote store rather buggy delete logic on Windows.

[sachinpkale]

Great! Thanks @mch2