testSyncedFlushWithConcurrentIndexing fails newly-introduced assertion

[DaveCTurner]

An assertion introduced in #39584 failed in CI:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+internalClusterTest/2543/console

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=523, name=elasticsearch[node_t1][generic][T#1], state=RUNNABLE, group=TGRP-FlushIT]
	at __randomizedtesting.SeedInfo.seed([B46A29F19373DDA9:399DAE3336803053]:0)
Caused by: java.lang.AssertionError: unexpected failure
	at __randomizedtesting.SeedInfo.seed([B46A29F19373DDA9]:0)
	at org.elasticsearch.action.support.replication.ReplicationOperation.onNoLongerPrimary(ReplicationOperation.java:218)
	at org.elasticsearch.action.support.replication.ReplicationOperation.access$700(ReplicationOperation.java:51)
	at org.elasticsearch.action.support.replication.ReplicationOperation$1.lambda$onFailure$2(ReplicationOperation.java:196)
	at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:70)
	at org.elasticsearch.action.ActionListener.onFailure(ActionListener.java:148)
	at org.elasticsearch.transport.TransportRequestDeduplicator$CompositeListener.onCompleted(TransportRequestDeduplicator.java:97)
	at org.elasticsearch.transport.TransportRequestDeduplicator$CompositeListener.onFailure(TransportRequestDeduplicator.java:111)
	at org.elasticsearch.cluster.action.shard.ShardStateAction$1.handleException(ShardStateAction.java:132)
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1118)
	at org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:660)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: SendRequestTransportException[[node_t1][127.0.0.1:43727][internal:cluster/shard/failure]]; nested: TransportException[TransportService is closed stopped can't send request];
	at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:639)
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:542)
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:517)
	at org.elasticsearch.cluster.action.shard.ShardStateAction.sendShardAction(ShardStateAction.java:118)
	at org.elasticsearch.cluster.action.shard.ShardStateAction.lambda$remoteShardFailed$0(ShardStateAction.java:169)
	at org.elasticsearch.transport.TransportRequestDeduplicator.executeOnce(TransportRequestDeduplicator.java:52)
	at org.elasticsearch.cluster.action.shard.ShardStateAction.remoteShardFailed(ShardStateAction.java:167)
	at org.elasticsearch.action.resync.TransportResyncReplicationAction$ResyncActionReplicasProxy.failShardIfNeeded(TransportResyncReplicationAction.java:213)
	at org.elasticsearch.action.support.replication.ReplicationOperation$1.onFailure(ReplicationOperation.java:195)
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59)
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1118)
	at org.elasticsearch.transport.TransportService$8.run(TransportService.java:960)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
	... 3 more
Caused by: TransportException[TransportService is closed stopped can't send request]
	at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:621)
	... 15 more

Not muting this as it looks like it's not specific to this one test.

[elasticmachine]

Pinging @elastic/es-distributed

[DaveCTurner]

Looks like testSyncedFlushWithConcurrentIndexing failed for a similar reason:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+internalClusterTest/2549/console

[DaveCTurner]

Also testAckedIndexing

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+internalClusterTest/2551/console

[DaveCTurner]

6.7 also affected?
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+internalClusterTest/1944/console

[dnhatn]

Thanks @DaveCTurner. This issue was resolved in #39723.