Failure in CloseWhileRelocatingShardsIT

[DaveCTurner]

I am seeing occasional failures of CloseWhileRelocatingShardsIT of the following form:

Suite: Test class org.elasticsearch.indices.state.CloseWhileRelocatingShardsIT
  2> liep. 25, 2019 9:37:59 PRIEŠPIET com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
  2> WARNING: Uncaught exception in thread: Thread[elasticsearch[node_sd4][generic][T#5],5,TGRP-CloseWhileRelocatingShardsIT]
  2> java.lang.AssertionError: max seq. no. [-1] does not match [207]
  2>    at __randomizedtesting.SeedInfo.seed([AE1B2D049A25A3A3]:0)
  2>    at org.elasticsearch.index.engine.ReadOnlyEngine.assertMaxSeqNoEqualsToGlobalCheckpoint(ReadOnlyEngine.java:153)
  2>    at org.elasticsearch.index.engine.ReadOnlyEngine.ensureMaxSeqNoEqualsToGlobalCheckpoint(ReadOnlyEngine.java:144)
  2>    at org.elasticsearch.index.engine.ReadOnlyEngine.<init>(ReadOnlyEngine.java:113)
  2>    at org.elasticsearch.index.engine.NoOpEngine.<init>(NoOpEngine.java:54)
  2>    at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1595)
  2>    at org.elasticsearch.index.shard.IndexShard.recoverLocallyUpToGlobalCheckpoint(IndexShard.java:1411)
  2>    at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:176)
  2>    at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:552)
  2>    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:769)
  2>    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
  2>    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>    at java.base/java.lang.Thread.run(Thread.java:835)

  2> REPRODUCE WITH: ./gradlew :server:integTest --tests "org.elasticsearch.indices.state.CloseWhileRelocatingShardsIT.testCloseWhileRelocatingShards" -Dtests.seed=AE1B2D049A25A3A3 -Dtests.security.manager=true -Dtests.jvms=4 -Dtests.locale=lt-LT -Dtests.timezone=America/Buenos_Aires -Dcompiler.java=12 -Druntime.java=12

Doesn't reproduce every time at 6275cd7 but normally fails after only a few iterations. Backing up to 69c94f4 before the merge of #43463 this test passes reliably (tens of successful iterations and counting).

Relates #41536 as this failure is only on the peer-recovery-retention-leases feature branch(es).

[elasticmachine]

Pinging @elastic/es-distributed

[DaveCTurner]

Muted in 96dd543.

[dnhatn]

Fixed in #44887