Move segment replication setting to IndexMetadata so that it can be set during index creation. #2284
Conversation
|
Can one of the admins verify this patch? |
|
I've cut an issue to fix the errors - #2285. I think this PR alone still holds value to run the tests against the general case as we iterate on segrep. |
| waitForDocs(additionalDocCount, indexer); | ||
| } | ||
| flush(INDEX_NAME); | ||
| Thread.sleep(1000); |
There was a problem hiding this comment.
Is this sleep to wait for flush operation to complete? And do we need to explicitly call flush?
There was a problem hiding this comment.
The sleep is to wait for the segment files to be copied over before we check the doc counts, without this we have a low risk that that process may not be complete. Yes, we call flush here so that we guarantee the primary is on a new commit point so we can test the replica can read it.
|
|
||
| import static org.opensearch.test.hamcrest.OpenSearchAssertions.assertHitCount; | ||
|
|
||
| @ClusterScope(scope = OpenSearchIntegTestCase.Scope.TEST, numDataNodes = 0) |
There was a problem hiding this comment.
Why is numDataNodes set to 0?
There was a problem hiding this comment.
This sets the number of nodes to start the test with. If unset a random number is used.
…et during index creation. This change also adds an integration test - SegmentReplicationIT. Signed-off-by: Marc Handalian <[email protected]>
|
❌ Gradle Check failure f4ca44f3ba7addefc6bf41b7f91d9551c331f0c1 |
|
❌ Gradle Check failure 438265f7050b8abe10d637a775c0aac47af5f9c0 |
| @@ -891,6 +882,10 @@ public int getNumberOfReplicas() { | |||
| return settings.getAsInt(IndexMetadata.SETTING_NUMBER_OF_REPLICAS, null); | |||
| } | |||
|
|
|||
| public boolean isSegrepEnabled() { | |||
There was a problem hiding this comment.
Nitpick - I'd prefer the method name to be explicit i.e. isSegmentReplicationEnabled
…et during index creation. (opensearch-project#2284) Signed-off-by: Marc Handalian <[email protected]>
…et during index creation. (opensearch-project#2284) Signed-off-by: Marc Handalian <[email protected]>
Description
Move segment replication setting to IndexMetadata so that it can be set during index creation. This change also adds a simple integration test - SegmentReplicationIT that indexes some docs then compares doc counts between primary and replica shards.
This integ test is failing 20% of the time and produces reproducible seeds. Example seed:
This failure is when comparing the doc count on line 58 of the test after performing a flush. I'm not sure whats going on here yet , reader.numDocs() is returning the correct number after performing a refresh in OpenSearchReaderManager.
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.