[Segment Replication] Fix flaky tests testSegmentReplicationStatsResponse() and testSegmentReplicationStatsWithTimeout() #6268

Rishikesh1159 · 2023-02-09T16:21:27Z

Signed-off-by: Rishikesh1159 [email protected]

Description

This PR :
-> Fixes flaky tests testSegmentReplicationStatsWithTimeout() and testSegmentReplicationStatsResponse().
-> Makes timingdata map to a concurrent map to handle concurrent modifications.
-> Moves timing data under detailed flag

Issues Resolved

Resolves #6261, #6255

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Rishikesh1159 <[email protected]>

github-actions · 2023-02-09T16:52:21Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/10932/
CommitID: 138c56b

server/src/internalClusterTest/java/org/opensearch/action/admin/ClientTimeoutIT.java

mch2 · 2023-02-10T04:04:17Z

server/src/main/java/org/opensearch/indices/replication/SegmentReplicationState.java

-    private final Map<String, Long> timingData;
+
+    // Timing data will have as many entries as stages, plus one
+    private final Map<String, Long> timingData = new ConcurrentHashMap<>(Stage.values().length + 1);


I think we still want to init this in constructors only and send this map over the wire, otherwise the map will empty on the other end when querying stats. You can make the map concurrent in the ctor or copy it to write.

I am going with copying the map and sending copy of map in output stream.

Signed-off-by: Rishikesh1159 <[email protected]>

github-actions · 2023-02-10T15:45:29Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationStatsIT.testSegmentReplicationStatsResponse

URL: https://build.ci.opensearch.org/job/gradle-check/10995/
CommitID: d15da00
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-02-10T16:57:34Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWithAReadOnlyBlock

URL: https://build.ci.opensearch.org/job/gradle-check/10999/
CommitID: d15da00
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov-commenter · 2023-02-10T16:59:43Z

Codecov Report

Merging #6268 (ba41b87) into main (1904158) will decrease coverage by 0.29%.
The diff coverage is 2.94%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #6268      +/-   ##
============================================
- Coverage     70.82%   70.53%   -0.29%     
+ Complexity    59078    58782     -296     
============================================
  Files          4799     4799              
  Lines        282434   282441       +7     
  Branches      40717    40716       -1     
============================================
- Hits         200041   199232     -809     
- Misses        65977    66780     +803     
- Partials      16416    16429      +13

Impacted Files	Coverage Δ
...cation/TransportSegmentReplicationStatsAction.java	`11.11% <0.00%> (ø)`
...st/action/cat/RestCatSegmentReplicationAction.java	`48.97% <0.00%> (-10.21%)`	⬇️
...h/indices/replication/SegmentReplicationState.java	`44.60% <4.16%> (-2.37%)`	⬇️
...search/aggregations/pipeline/ParsedDerivative.java	`0.00% <0.00%> (-79.17%)`	⬇️
...main/java/org/opensearch/common/network/Cidrs.java	`0.00% <0.00%> (-77.78%)`	⬇️
...in/java/org/opensearch/test/MockKeywordPlugin.java	`0.00% <0.00%> (-75.00%)`	⬇️
...rch/search/query/QueryPhaseExecutionException.java	`0.00% <0.00%> (-66.67%)`	⬇️
...n/admin/cluster/node/tasks/get/GetTaskRequest.java	`30.30% <0.00%> (-63.64%)`	⬇️
...a/org/opensearch/client/cluster/ProxyModeInfo.java	`0.00% <0.00%> (-60.00%)`	⬇️
...rg/apache/lucene/queries/SpanMatchNoDocsQuery.java	`42.85% <0.00%> (-57.15%)`	⬇️
... and 524 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: Rishikesh1159 <[email protected]>

github-actions · 2023-02-11T00:15:00Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/11034/
CommitID: 07fc9c3

Signed-off-by: Rishikesh1159 <[email protected]>

github-actions · 2023-02-15T17:52:15Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/11187/
CommitID: a18d05f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-02-15T18:12:37Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/11189/
CommitID: 95f2501

dreamer-89 · 2023-02-20T20:42:12Z

...c/internalClusterTest/java/org/opensearch/indices/replication/SegmentReplicationStatsIT.java

-        // index 10 docs
-        for (int i = 0; i < 10; i++) {
-            client().prepareIndex(INDEX_NAME).setId(Integer.toString(i)).setSource("field", "value" + i).execute().actionGet();
+    public void testSegmentReplicationStatsResponse() {


@Rishikesh1159 : I also fixed this flaky test as part of #6370. Please rebase your changes against main ?

Sure let me rebase

FYI, I see one run where this is still flaky #6366 (comment).
Previously, I root caused the problem happening due to race condition and fixed by asserting on replication state to be done which fixed it from 20% failing -> 0%. The assertBusy block by default waits for 10s not sure if segment replication needs more time to complete. May be we can increase the assertBusy timeout to 60secs and check if this test is still flaky.

Yes testSegmentReplicationStatsResponse() was flaky previously because of stage not reaching done within certain time. But now I have changed the test completely. As we are only testing API response and we are concerned only about API response but not about segment replication stage. So I changed the test and we no more assert on segrep stage which was causing flakiness.

With changes in testSegmentReplicationStatsResponse() we only assert if we get a successful or failure response from API call. We are not concerned about if segment replication event has finished processing or not, as purpose of API is only to give response irrespective of state of segrep event.

I think there is still value in asserting the replication state as it can uncover bugs where replication never completes for various reasons and it is also part of API response. I will leave it for you to decide.

Sure, I can add asserts on replication state. It doesn't break anything but as you said there is a chance it can uncover some bugs. I have added that in latest commit. Thanks @dreamer-89

github-actions · 2023-02-20T21:22:47Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/11433/
CommitID: 043d886

Signed-off-by: Rishikesh1159 <[email protected]>

github-actions · 2023-02-20T23:08:40Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/11444/
CommitID: e9b91d0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Rishikesh1159 <[email protected]>

github-actions · 2023-02-21T00:00:00Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/11447/
CommitID: bb0f624

dreamer-89 · 2023-02-21T03:19:13Z

...c/internalClusterTest/java/org/opensearch/indices/replication/SegmentReplicationStatsIT.java

@@ -121,6 +131,10 @@ public void testSegmentReplicationStatsResponseForActiveAndCompletedOnly() throw
            completedOnlyResponse.shardSegmentReplicationStates().get(INDEX_NAME).get(0).getStage(),
            SegmentReplicationState.Stage.DONE
        );
+        assertThat(


We can use assertEquals instead of assertThat here. This is applicable for other usages of assertThat as well

Sure I will update all usages

dreamer-89 · 2023-02-21T03:20:45Z

...c/internalClusterTest/java/org/opensearch/indices/replication/SegmentReplicationStatsIT.java

-        // index 10 docs
-        for (int i = 0; i < 10; i++) {
-            client().prepareIndex(INDEX_NAME).setId(Integer.toString(i)).setSource("field", "value" + i).execute().actionGet();
+    public void testSegmentReplicationStatsResponse() {


I think there is still value in asserting the replication state as it can uncover bugs where replication never completes for various reasons and it is also part of API response. I will leave it for you to decide.

Signed-off-by: Rishikesh1159 <[email protected]>

github-actions · 2023-02-21T18:43:21Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/11515/
CommitID: 6950d71

dreamer-89 · 2023-02-21T18:51:03Z

.../org/opensearch/action/admin/indices/replication/TransportSegmentReplicationStatsAction.java

@@ -140,7 +140,7 @@ protected SegmentReplicationState shardOperation(SegmentReplicationStatsRequest
            singleIndexWithSegmentReplicationDisabled = shardRouting.getIndexName();
            return null;
        }
-        if (indexShard.indexSettings().isSegRepEnabled() == false) {
+        if (indexShard.indexSettings().isSegRepEnabled() == false || shardRouting.primary()) {


Why this change is needed ?

If shard is primary we don't even need to make a call to get segment replication state, as segment replication only happens on replica shards. If we don't have this primary check here, it will make a call to get segment replication state and check some collections and eventually return null on a primary shard. So with this addition of condition check we are just returning null early if it is a primary sahrd

dreamer-89 · 2023-02-21T18:54:20Z

...c/internalClusterTest/java/org/opensearch/indices/replication/SegmentReplicationStatsIT.java

        }
        refresh(INDEX_NAME);
-        waitForSearchableDocs(10L, asList(primaryNode, replicaNode));
+        ensureSearchable(INDEX_NAME);

        assertBusy(() -> {


As I mentioned here, seen one instance where this test failed. May be we can increase the assertBusy timeout and see if it helps.

ah sure. May be I can increase it to 60sec just to be certain that replication has completed as you said before.

Signed-off-by: Rishikesh1159 <[email protected]>

github-actions · 2023-02-21T20:24:45Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      2 org.opensearch.cluster.service.MasterServiceTests.classMethod
      1 org.opensearch.cluster.service.MasterServiceTests.testThrottlingForMultipleTaskTypes

URL: https://build.ci.opensearch.org/job/gradle-check/11531/
CommitID: ba41b87
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Rishikesh1159 · 2023-02-21T22:11:53Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕

TEST FAILURES:
      2 org.opensearch.cluster.service.MasterServiceTests.classMethod
      1 org.opensearch.cluster.service.MasterServiceTests.testThrottlingForMultipleTaskTypes
URL: https://build.ci.opensearch.org/job/gradle-check/11531/

CommitID: ba41b87
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Issue already opened for these flaky tests #5958

Fix flaky tests.

138c56b

Signed-off-by: Rishikesh1159 <[email protected]>

Rishikesh1159 requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, adnapibar, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng, VachaShah and xuezhou25 as code owners February 9, 2023 16:21

Rishikesh1159 added the skip-changelog label Feb 9, 2023

mch2 reviewed Feb 9, 2023

View reviewed changes

server/src/internalClusterTest/java/org/opensearch/action/admin/ClientTimeoutIT.java Outdated Show resolved Hide resolved

dreamer-89 reviewed Feb 9, 2023

View reviewed changes

server/src/internalClusterTest/java/org/opensearch/action/admin/ClientTimeoutIT.java Show resolved Hide resolved

mch2 reviewed Feb 10, 2023

View reviewed changes

Remove unnecessary tests and fix tests.

d15da00

Signed-off-by: Rishikesh1159 <[email protected]>

Update testSegmentReplicationStatsResponse().

07fc9c3

Signed-off-by: Rishikesh1159 <[email protected]>

Rishikesh1159 added 2 commits February 15, 2023 17:30

Fix falky tests and add concurrent map in SegmentReplicationState.

a18d05f

Signed-off-by: Rishikesh1159 <[email protected]>

Spotless Apply

95f2501

Signed-off-by: Rishikesh1159 <[email protected]>

dreamer-89 reviewed Feb 20, 2023

View reviewed changes

Merge branch 'main' into fix-flaky-test

043d886

moving timing data under detailed flag.

e9b91d0

Signed-off-by: Rishikesh1159 <[email protected]>

spotless apply

bb0f624

Signed-off-by: Rishikesh1159 <[email protected]>

dreamer-89 reviewed Feb 21, 2023

View reviewed changes

Add assert busy and some code refactoring.

6950d71

Signed-off-by: Rishikesh1159 <[email protected]>

dreamer-89 reviewed Feb 21, 2023

View reviewed changes

Increasing assertBusy Timeout

ba41b87

Signed-off-by: Rishikesh1159 <[email protected]>

dreamer-89 approved these changes Feb 21, 2023

View reviewed changes

Rishikesh1159 merged commit 4316f96 into opensearch-project:main Feb 21, 2023

Rishikesh1159 mentioned this pull request Feb 22, 2023

[Backport 2.x] [Segment Replication] Backport of #6268 #6426

Merged

6 tasks

Rishikesh1159 mentioned this pull request Mar 19, 2023

[BUG] Flaky test org.opensearch.action.admin.ClientTimeoutIT.testSegmentReplicationStatsWithTimeout #6255

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Segment Replication] Fix flaky tests testSegmentReplicationStatsResponse() and testSegmentReplicationStatsWithTimeout() #6268

[Segment Replication] Fix flaky tests testSegmentReplicationStatsResponse() and testSegmentReplicationStatsWithTimeout() #6268

Rishikesh1159 commented Feb 9, 2023 •

edited

Loading

github-actions bot commented Feb 9, 2023

mch2 Feb 10, 2023

Rishikesh1159 Feb 10, 2023

github-actions bot commented Feb 10, 2023

github-actions bot commented Feb 10, 2023

codecov-commenter commented Feb 10, 2023 •

edited

Loading

github-actions bot commented Feb 11, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

dreamer-89 Feb 20, 2023 •

edited

Loading

Rishikesh1159 Feb 20, 2023

dreamer-89 Feb 21, 2023 •

edited

Loading

Rishikesh1159 Feb 21, 2023

Rishikesh1159 Feb 21, 2023 •

edited

Loading

dreamer-89 Feb 21, 2023

Rishikesh1159 Feb 21, 2023

github-actions bot commented Feb 20, 2023

github-actions bot commented Feb 20, 2023

github-actions bot commented Feb 21, 2023

dreamer-89 Feb 21, 2023

Rishikesh1159 Feb 21, 2023

dreamer-89 Feb 21, 2023

github-actions bot commented Feb 21, 2023

dreamer-89 Feb 21, 2023

Rishikesh1159 Feb 21, 2023

dreamer-89 Feb 21, 2023

Rishikesh1159 Feb 21, 2023 •

edited

Loading

github-actions bot commented Feb 21, 2023

Rishikesh1159 commented Feb 21, 2023

Gradle Check (Jenkins) Run Completed with:

[Segment Replication] Fix flaky tests testSegmentReplicationStatsResponse() and testSegmentReplicationStatsWithTimeout() #6268

[Segment Replication] Fix flaky tests testSegmentReplicationStatsResponse() and testSegmentReplicationStatsWithTimeout() #6268

Conversation

Rishikesh1159 commented Feb 9, 2023 • edited Loading

Description

Issues Resolved

Check List

github-actions bot commented Feb 9, 2023

Gradle Check (Jenkins) Run Completed with:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Feb 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Feb 10, 2023

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Feb 10, 2023 • edited Loading

Codecov Report

github-actions bot commented Feb 11, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Feb 15, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Feb 15, 2023

Gradle Check (Jenkins) Run Completed with:

dreamer-89 Feb 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dreamer-89 Feb 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rishikesh1159 Feb 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Feb 20, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Feb 20, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Feb 21, 2023

Gradle Check (Jenkins) Run Completed with:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Feb 21, 2023

Gradle Check (Jenkins) Run Completed with:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rishikesh1159 Feb 21, 2023 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Feb 21, 2023

Gradle Check (Jenkins) Run Completed with:

Rishikesh1159 commented Feb 21, 2023

Gradle Check (Jenkins) Run Completed with:

Rishikesh1159 commented Feb 9, 2023 •

edited

Loading

codecov-commenter commented Feb 10, 2023 •

edited

Loading

dreamer-89 Feb 20, 2023 •

edited

Loading

dreamer-89 Feb 21, 2023 •

edited

Loading

Rishikesh1159 Feb 21, 2023 •

edited

Loading

Rishikesh1159 Feb 21, 2023 •

edited

Loading