Skip to content

Commit

Permalink
Fix Test Failure in testMasterFailOverWithQueuedDeletes (#76865)
Browse files Browse the repository at this point in the history
We can have a freak situation here where running the get snapshots
request concurrently with a delete produces a missing snapshot exception
if a snasphot is deleted just as its metadata is fetched.
This is a known issue and a fix is tricky (SLM etc. work around this issue
in tests and prod code by using the ignore-unavailable flag for example).
In this test we can easily fix the problem by just using the deterministic waiting
on cluster state before asserting that the snapshots are gone from the repo.

closes #76549
  • Loading branch information
original-brownbear authored Aug 24, 2021
1 parent b3c1170 commit 98bf4eb
Showing 1 changed file with 2 additions and 9 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@
import org.elasticsearch.action.support.GroupedActionListener;
import org.elasticsearch.action.support.PlainActionFuture;
import org.elasticsearch.action.support.master.AcknowledgedResponse;
import org.elasticsearch.cluster.ClusterState;
import org.elasticsearch.cluster.SnapshotDeletionsInProgress;
import org.elasticsearch.cluster.SnapshotsInProgress;
import org.elasticsearch.common.Strings;
Expand Down Expand Up @@ -499,14 +498,8 @@ public void testMasterFailOverWithQueuedDeletes() throws Exception {
expectThrows(SnapshotException.class, snapshotThreeFuture::actionGet);

logger.info("--> verify that all snapshots are gone and no more work is left in the cluster state");
assertBusy(() -> {
assertThat(client().admin().cluster().prepareGetSnapshots(repoName).get().getSnapshots(), empty());
final ClusterState state = clusterService().state();
final SnapshotsInProgress snapshotsInProgress = state.custom(SnapshotsInProgress.TYPE);
assertThat(snapshotsInProgress.entries(), empty());
final SnapshotDeletionsInProgress snapshotDeletionsInProgress = state.custom(SnapshotDeletionsInProgress.TYPE);
assertThat(snapshotDeletionsInProgress.getEntries(), empty());
}, 30L, TimeUnit.SECONDS);
awaitNoMoreRunningOperations();
assertThat(client().admin().cluster().prepareGetSnapshots(repoName).get().getSnapshots(), empty());
}

public void testAssertMultipleSnapshotsAndPrimaryFailOver() throws Exception {
Expand Down

0 comments on commit 98bf4eb

Please sign in to comment.